--- title: "Only 29.7% of Our Homepage’s Claimed AI-Bot Hits Were Verified" description: "groas’s 30-day edge measurement found a wide gap between claimed AI-bot traffic and verified live-answer fetches. Here is what an agency AEO report should show instead." url: "https://groas.com/post/only-30-of-the-ai-bot-hits-on-our-homepa" image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/55204b8c-b80b-4144-a1a3-f0703f45fa8b.png" published: "2026-10-18T05:19:09.796Z" modified: "2026-10-18T05:19:09.868Z" --- October 18, 2026 · 10 min read # Only 29.7% of Our Homepage’s Claimed AI-Bot Hits Were Verified [Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/) ![Clay doorman inspects a queue all wearing 'GPTBot' name tags; only three have official seals, the rest are scrawled fakes in clumsy disguises.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/55204b8c-b80b-4144-a1a3-f0703f45fa8b.png) In this article 1. [What the 30-day edge measurement counted](#what-the-30-day-edge-measurement-counted) 2. [Finding 1: the busiest paths had the weakest identity checks](#finding-1-the-busiest-paths-had-the-weakest-identity-checks) 3. [Finding 2: the technical-content group reversed the pattern](#finding-2-the-technical-content-group-reversed-the-pattern) 4. [Finding 3: category pages were crawled, but not fetched for live answers](#finding-3-category-pages-were-crawled-but-not-fetched-for-live-answers) 5. [Finding 4: verification and retrieval are separate jobs](#finding-4-verification-and-retrieval-are-separate-jobs) 6. [What a raw-count client report gets wrong](#what-a-raw-count-client-report-gets-wrong) 7. [Put verified live-answer fetches per page in the report](#put-verified-live-answer-fetches-per-page-in-the-report) Only **29.7%** of the requests claiming an AI-crawler identity on the groas homepage passed network verification. Over 30 days, the edge recorded 6,506 such requests and verified 1,930. On `/robots.txt`, 3,912 of 13,494 passed: **29.0%**. If an agency reports raw user-agent counts as AI attention, it can show a client more than three times as much activity on those paths as the verified count supports. That does not mean every unverified request was a malicious scraper. It means the logs could not establish that the request came from the vendor named in its header. Nor does a verified crawl mean an answer engine showed a client’s page to anyone. Those distinctions are the whole story. I spent years in paid search watching reports confuse impressions with progress. A chart goes up; qualified pipeline does not; somebody calls the chart _top-of-funnel momentum_. AI visibility is getting its own version of that routine. At [groas](https://groas.com/), we measured incoming requests at the edge and separated claimed bot identity, verified identity, and live-answer retrieval. The useful reporting question is not “How many bots hit the site?” It is **which pages received verified live-answer fetches**. ## What the 30-day edge measurement counted Browser analytics cannot give you this picture. A crawler request may never run a client-side tag, and a CDN can serve a cached page without passing the request through a CMS plugin. [Client-side tracking has that blind spot](https://www.reddit.com/r/TechSEO/comments/1wb7slo/three_bugs_i_found_in_my_own_aicrawler_checking/). Our measurement logged incoming HTTP requests at the edge, before an origin application or cache could hide them from site-level tracking. These are **groas’s own edge measurements**, not an independently published, site-wide benchmark. The table covers the primary paths grouped in this analysis over one 30-day window. It tells us what happened in those logs, not what percentage of AI traffic every other site should expect. ### Three counts that should never share one label We classified the requests in three tiers: 1. **`hits`:** Requests whose `User-Agent` header claimed an AI-bot identity, including strings such as `GPTBot`, `ChatGPT-User`, `ClaudeBot`, `PerplexityBot`, and `OAI-SearchBot`. A header is a claim, not proof; anyone can write one into a request. 2. **`verifiedHits`:** Claimed bot requests that passed the applicable network check: a connecting IP matched a vendor-published prefix, or the request passed Forward-Confirmed Reverse DNS (FCrDNS) where that method applied. This follows the distinction behind [verified-bot checks](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/): establish who connected rather than trusting what the header says. 3. **`liveAnswerHits`:** Verified requests classified as live, user-directed retrieval rather than background crawling. [OpenAI distinguishes its crawler and user-triggered agents](https://geoiphub.com/blog/gptbot-vs-oai-searchbot-vs-chatgpt-user). A fetch shows that a page was requested in a live-answer workflow. It does **not** establish that the page was cited, recommended, or viewed by a buyer. ![Three tiers of traffic measurement: claimed AI-bot hits, network-verified hits, and verified live-answer fetches.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/a87e1791-ce0c-4978-b520-465eceda59f5.png) | Site path or asset group | Raw `hits` | Network-verified `verifiedHits` | Verification rate | `liveAnswerHits` | | :-------------------------------- | ---------: | ------------------------------: | ----------------: | ---------------: | | `/robots.txt` | 13,494 | 3,912 | 29.0% | 0 | | Homepage (`/`) | 6,506 | 1,930 | 29.7% | 48 | | Feature and category pages | 1,842 | 254 | 13.8% | 0 | | Changelog and technical teardowns | 414 | 402 | 97.1% | 402 | Across these rows, the edge recorded **22,256 claimed AI-bot hits, 6,498 verified hits, and 450 classified live-answer fetches**. The last figure is about 2% of the raw count. It is not a conversion rate: these are different kinds of request, and the logs do not connect them to leads or revenue. The table also groups changelog and technical teardown material. It does not provide a page-by-page breakdown within that row. I would not turn its 402 live-answer fetches into a claim about one post without that breakdown. The contrast between path groups is still sharp enough to matter. ## Finding 1: the busiest paths had the weakest identity checks `/robots.txt` attracted the most claimed AI-bot traffic: 13,494 requests. The homepage followed with 6,506. Yet **9,582 requests to `/robots.txt` and 4,576 to the homepage did not pass verification**. If a dashboard simply searches logs for recognizable bot names, all of those requests can end up under an OpenAI or Anthropic logo. There is a plausible operational reason these paths draw noisy traffic. Automated tools often start at `/robots.txt` or the root domain before moving deeper into a site. A scraper can also present a trusted-looking user agent. In [a review of crawler behavior](https://www.reddit.com/r/TechSEO/comments/1vy4re8/most_ai_crawlers_hitting_your_site_cannot_be/), verifiable identity was far from universal. But these edge counts alone do not identify the operator or purpose of each failed request. **“Unverified” is the finding; “spoofed scraper” is not a finding for every row.** That limit makes the reporting decision easier, not harder. When roughly seven in ten claimed hits on the first paths a client will inspect lack verified identity, raw counts do not belong on a slide labeled _AI attention_. ## Finding 2: the technical-content group reversed the pattern The changelog and technical teardown group recorded only 414 claimed AI-bot hits. **Of those, 402 passed verification, a 97.1% rate; all 402 were classified as live-answer fetches.** Set that beside the homepage: 6,506 raw hits, 1,930 verified hits, and 48 live-answer fetches. The smaller row carries the stronger retrieval signal. That is the comparison I would want in a client report. It distinguishes a page group requested during live-answer activity from a homepage repeatedly hit by requests with unproven identities. It also gives an agency something specific to examine: which material is being retrieved, and what on those pages answers the kinds of questions people ask. It does not prove that technical teardowns always outperform other content. We have one 30-day measurement, and the table does not isolate an individual post or show the prompts behind the fetches. It does show why publishing generic FAQ material and then celebrating a rising site-wide bot count is a poor substitute for looking at **verified retrieval by page**. If the page-level export is available, inspect it. If it is not, do not pretend the grouped row is a page-level result. ## Finding 3: category pages were crawled, but not fetched for live answers Feature and category pages produced **1,842 claimed hits, 254 verified hits, and zero classified live-answer fetches** in the window. Their verification rate, 13.8%, was the lowest in the table. Background crawler activity and live retrieval are plainly not interchangeable here. An agency could still report the 254 verified requests as evidence that known crawlers reached those pages. It could not fairly describe them as 254 buyer interactions, recommendations, or citations. And it should not describe the other 1,588 as proven competitor scraping. The measurement did not establish that either. Zero live-answer fetches is worth investigating, especially if those pages matter commercially. It is not proof that nobody asked an answer engine about the products or that the pages have no value. This measurement records fetches reaching these paths during this window. **It does not observe every prompt or every answer.** ## Finding 4: verification and retrieval are separate jobs The agent named in a request matters, but the name alone is not a measurement method. [FCrDNS works for some major crawlers, while others require checks against published IP ranges](https://eseospace.com/blog/verify-ai-bots-in-server-logs/). A tracker that relies only on a user-agent string can accept an impostor. A tracker that applies the wrong identity check can miss a genuine request. After identity comes a second question: was the verified agent doing background crawling or live, user-directed retrieval? The draft temptation is to go one step further and assign a neat editorial preference to each platform: this agent likes pricing, that one likes workflows. The table cannot support that split. It contains no per-platform content breakdown. For this analysis, the defensible conclusion is narrower and more useful: **verify the caller, then classify the fetch**. Keep the two operations visible in the report. ![Hollow bot figures approach a site facade while a verified data stream reaches an answer engine.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/22d5da25-2aa1-4ba8-8fa9-bfd871b874b4.png) ## What a raw-count client report gets wrong On the homepage, a raw user-agent dashboard could show **6,506 “AI-bot visits”** where the edge verified 1,930 bot requests and classified 48 live-answer fetches. On `/robots.txt`, it could show 13,494 where 3,912 passed verification and none were classified as live-answer fetches. The raw-to-verified overcount is about **3.3 times on the homepage and 3.4 times on `/robots.txt`**. Those are different problems, not progressively smaller versions of the same metric. The first gap concerns **identity**: did the request come from the claimed bot? The second concerns **intent**: did a verified request belong to live-answer retrieval rather than background crawling? Neither count, on its own, tells us whether a page appeared in an answer or produced a lead. This matters because clients naturally check their homepage and core commercial pages first. A large number beside the root domain looks persuasive. It is also where this measurement found a great deal of unverified traffic. Put the raw number in a presentation without its verification rate and you make the least trustworthy part of the dataset look like the headline result. I have seen the paid-search version: impressions rise, pipeline stays flat, and the explanation becomes _awareness_. Sometimes awareness is a legitimate objective. Sometimes the report has simply promoted an easy-to-count event into a claim it cannot carry. Do not make a client discover the difference after paying for it. ## Put verified live-answer fetches per page in the report For an agency selling AI visibility, **verified live-answer fetches per page** is the crawler metric I would put in the executive report. Show the raw and verified counts in the supporting detail, alongside the verification method and the measurement window. Do not relabel a fetch as a citation or a sale. If the client needs commercial accountability, examine the fetches next to the business outcomes the agency already tracks, rather than asking crawler traffic to stand in for revenue. At [groas](https://groas.com/), the point of logging these signals is to make the work and its limits visible, not to dress every request up as demand. [groas for agencies](https://groas.com/for-agencies) is built around white-label paid and organic search execution with logged actions and performance signals. That is a more useful basis for a retainer conversation than a chart of headers anyone can claim. ![An inflated raw-crawler count is crossed out beside a report of verified live-answer fetches.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/10d5a936-e7d8-4d9f-af6b-ee501d6659c0.png) ### Four questions before buying a white-label AEO dashboard Some [AI-visibility tools](https://www.brandlight.ai/blog/best-ai-visibility-tools-for-agencies) concentrate on prompt tracking and reporting rather than site execution. That may be the product you need, but the sales demo should not blur measurement, verification, and implementation. Ask: 1. **How do you verify identity?** Do you match connecting IPs against vendor-published ranges or use FCrDNS where appropriate, or do you filter only on `User-Agent`? 2. **Can you separate crawl purpose?** Does the report distinguish background agents such as `GPTBot` from verified, live user-directed retrieval such as `ChatGPT-User`? 3. **Where do you measure requests?** Will the system see a bot request served by the CDN without reaching the CMS or its analytics plugin? 4. **Who does the work?** Does the platform make and log changes, or hand the agency a list of recommendations to implement elsewhere? A vendor should be able to answer the first three without replacing the question with a slide about _AI presence_. The fourth tells you whether you are buying execution or another dashboard your team must interpret and act on. ### Measure before the CMS loses the request This is partly an infrastructure problem. Agencies manage sites on different CMSs and hosting stacks. A browser tag does not measure a request that never runs browser JavaScript. A server-side plugin may not see a request satisfied at the CDN. Edge-level logging puts the observation point before that handoff, so the same incoming request can be checked for claimed identity, network identity, and retrieval classification. That does not make every downstream question disappear. The edge cannot tell you whether a model cited a page in its final answer or whether a customer bought because of it. It can give you a cleaner account of **what reached the site and which requests passed verification**. That is a sound place to start, and a much better place to stop making claims the logs cannot defend. The decision supported by these 30 days is straightforward. Do not renew an AEO reporting arrangement because a dashboard found thousands of bot-shaped headers on a homepage. Ask for verified live-answer fetches by page, with raw counts and identity checks available underneath. If the platform cannot separate those numbers, it is not measuring the result it says it is measuring. ## Frequently Asked Questions ### What is the difference between a claimed AI-bot hit and a network-verified hit? A claimed hit is any request that uses a User-Agent header to claim an AI-bot identity. A verified hit is a claimed bot request that successfully passed a network check, meaning the connecting IP matched a vendor-published prefix or passed Forward-Confirmed Reverse DNS. ### Why are raw AI-bot counts reported from user agents misleading? Raw user-agent counts can be highly misleading because a header is just a claim, not proof. This can lead to an overcount of AI activity, which was found to be about 3.3 times on the homepage and 3.4 times on /robots.txt. ### Why is standard browser analytics insufficient for measuring AI traffic? Browser analytics can miss crawler requests that never run a client-side tag or when a CDN serves a cached page. Edge-level logging records incoming HTTP requests before an origin application or cache can hide them from site-level tracking. ### Can a verified request prove a page was cited by an AI or resulted in a lead? No, the measurement only tracks verified requests classified as live-answer retrieval. These logs do not connect requests to leads, revenue, or establish whether an answer engine cited a page in its final response. ### How does content type affect the quality of AI traffic detection? Content like changelog and technical teardowns recorded a 97.1% verification rate, meaning nearly all requests were verified and classified as live-answer fetches. In contrast, the homepage had a lower verification rate of 29.7%. ### What should agencies report instead of raw bot traffic counts? Agencies should report verified live-answer fetches per page in the executive report. Raw and verified counts should be provided in the supporting detail, showing the verification method and measurement window. ## Pay For Results, Not For Hours Businesses buy the outcome, agencies resell it, and groas answers for it either way. [See If You Qualify](https://groas.typeform.com/to/xC1bQNUT) ## Structured data ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/only-30-of-the-ai-bot-hits-on-our-homepa#webpage","url":"https://groas.com/post/only-30-of-the-ai-bot-hits-on-our-homepa","name":"Only 29.7% of Our Homepage’s Claimed AI-Bot Hits Were Verified","description":"groas’s 30-day edge measurement found a wide gap between claimed AI-bot traffic and verified live-answer fetches. Here is what an agency AEO report should show instead.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/55204b8c-b80b-4144-a1a3-f0703f45fa8b.png"},"breadcrumb":{"@id":"https://groas.com/post/only-30-of-the-ai-bot-hits-on-our-homepa#breadcrumb"},"dateModified":"2026-10-18T05:19:09.868Z","mainEntity":{"@id":"https://groas.com/post/only-30-of-the-ai-bot-hits-on-our-homepa#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/only-30-of-the-ai-bot-hits-on-our-homepa#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Only 29.7% of Our Homepage’s Claimed AI-Bot Hits Were Verified","item":"https://groas.com/post/only-30-of-the-ai-bot-hits-on-our-homepa"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/only-30-of-the-ai-bot-hits-on-our-homepa#article","url":"https://groas.com/post/only-30-of-the-ai-bot-hits-on-our-homepa","mainEntityOfPage":{"@id":"https://groas.com/post/only-30-of-the-ai-bot-hits-on-our-homepa#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Only 29.7% of Our Homepage’s Claimed AI-Bot Hits Were Verified","description":"groas’s 30-day edge measurement found a wide gap between claimed AI-bot traffic and verified live-answer fetches. Here is what an agency AEO report should show instead.","datePublished":"2026-10-18T05:19:09.796Z","dateModified":"2026-10-18T05:19:09.868Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/55204b8c-b80b-4144-a1a3-f0703f45fa8b.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"verified, claimed, live-answer fetches, live-answer, hits, fetches, claimed ai-bot, ai-bot","wordCount":2034,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/only-30-of-the-ai-bot-hits-on-our-homepa#faq","mainEntity":[{"@type":"Question","name":"What is the difference between a claimed AI-bot hit and a network-verified hit?","acceptedAnswer":{"@type":"Answer","text":"A claimed hit is any request that uses a User-Agent header to claim an AI-bot identity. A verified hit is a claimed bot request that successfully passed a network check, meaning the connecting IP matched a vendor-published prefix or passed Forward-Confirmed Reverse DNS."}},{"@type":"Question","name":"Why are raw AI-bot counts reported from user agents misleading?","acceptedAnswer":{"@type":"Answer","text":"Raw user-agent counts can be highly misleading because a header is just a claim, not proof. This can lead to an overcount of AI activity, which was found to be about 3.3 times on the homepage and 3.4 times on /robots.txt."}},{"@type":"Question","name":"Why is standard browser analytics insufficient for measuring AI traffic?","acceptedAnswer":{"@type":"Answer","text":"Browser analytics can miss crawler requests that never run a client-side tag or when a CDN serves a cached page. Edge-level logging records incoming HTTP requests before an origin application or cache can hide them from site-level tracking."}},{"@type":"Question","name":"Can a verified request prove a page was cited by an AI or resulted in a lead?","acceptedAnswer":{"@type":"Answer","text":"No, the measurement only tracks verified requests classified as live-answer retrieval. These logs do not connect requests to leads, revenue, or establish whether an answer engine cited a page in its final response."}},{"@type":"Question","name":"How does content type affect the quality of AI traffic detection?","acceptedAnswer":{"@type":"Answer","text":"Content like changelog and technical teardowns recorded a 97.1% verification rate, meaning nearly all requests were verified and classified as live-answer fetches. In contrast, the homepage had a lower verification rate of 29.7%."}},{"@type":"Question","name":"What should agencies report instead of raw bot traffic counts?","acceptedAnswer":{"@type":"Answer","text":"Agencies should report verified live-answer fetches per page in the executive report. Raw and verified counts should be provided in the supporting detail, showing the verification method and measurement window."}}]}]} ```