---
title: "We Logged 30 Days of AI Bot Traffic. The Useful Number Wasn’t Bot Hits"
description: "In groas’s 30-day edge logs, only 26% of claimed AI-bot fetches to robots.txt were verified. One dated post drew 417 user-triggered fetches. Here’s what to measure instead."
url: "https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-4"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/5d0c50c0-b85e-492f-84f8-12085b7475db.png"
published: "2026-10-18T05:34:47.199Z"
modified: "2026-10-18T05:34:47.278Z"
---

October 18, 2026 · 10 min read

# We Logged 30 Days of AI Bot Traffic. The Useful Number Wasn’t Bot Hits

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![A bouncer's UV torch exposes a forged 'BOT' name tag while a long queue of identical 'BOT' guests waits in the rain, suggesting most bot hits are fakes.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/5d0c50c0-b85e-492f-84f8-12085b7475db.png)

In this article

1. [The dataset separates a claim from a page fetch](#the-dataset-separates-a-claim-from-a-page-fetch)
2. [The biggest bot-hit total mostly measured noise](#the-biggest-bot-hit-total-mostly-measured-noise)
3. [The homepage gets checked; a dated post gets fetched](#the-homepage-gets-checked-a-dated-post-gets-fetched)
4. [Pair page-fetch counts with what assistants actually say](#pair-page-fetch-counts-with-what-assistants-actually-say)
5. [Test your own pages without counting permission checks as wins](#test-your-own-pages-without-counting-permission-checks-as-wins)
6. [Put the page-level number in next month’s report](#put-the-page-level-number-in-next-months-report)

Something claiming to be an AI bot asked groas.com for `/robots.txt` **15,093 times in 30 days**. Our edge logs show that only **3,972 of those requests, or 26%,** came from IPs in the operators’ published ranges. The rest could make the claim in a user-agent string; they could not back it up.

That gap makes total bot hits a poor AI-visibility metric. A request to check crawl permissions is not a request for an answer page, either. To find the useful signal, I split the logs into claimed hits, verified hits, and verified, user-triggered fetches of actual pages. The last group gets closest to showing which pages an assistant might use in a live response. It still does not prove what the assistant ultimately said or cited.

## The dataset separates a claim from a page fetch

I pulled 30 days of edge logs for groas.com and sorted requests claiming an AI-bot identity into three tiers. The distinction matters because the same dashboard total can contain impersonators, routine crawlers, permission checks, and fetches prompted by a person asking an assistant a question.

| Tier                | What I counted                                                                                                  | What the count can tell you                                                                                        |
| ------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Claimed hits        | Requests with an AI-bot name, such as `ChatGPT-User`, `ClaudeBot`, or `PerplexityBot`, in the user-agent string | What the requester _said_ it was                                                                                   |
| Verified hits       | Claimed requests whose source IP matched that operator’s published range                                        | Which claimed requests passed an identity check                                                                    |
| Live-answer fetches | Verified, user-triggered fetches of an actual page URL, such as those made by ChatGPT-User or Claude-User       | Which pages were fetched in connection with live assistant activity, not whether they appeared in the final answer |

**A user-agent is a claim; the source IP is the first check.** [Digimirror describes the same claim-versus-proof distinction](https://digimirror.io/your-bot-traffic-logs-are-lying-to-you/). In one new-domain audit, it found that [only 6 of 33 requests claiming to be live assistant fetches were real](https://digimirror.io/your-bot-traffic-logs-are-lying-to-you/); only 107 of 799 Googlebot-labeled requests came from verified Google IPs. [Promptwatch counted about 18% less AI-crawler traffic on average after IP-range verification](https://promptwatch.com/blog/new-feature-verified-ai-crawler-data). And [CHEQ puts false representation at 1 in 7 invalid retail visits](https://cheq.ai/blog/agentic-commerce-on-black-friday-cyber-monday/), describing scrapers that wear known crawler names while real agents can look like ordinary Chrome traffic.

Those are different datasets with different denominators, not a pooled industry rate. They do explain why I stopped accepting a bot name in a log as evidence of AI interest. I used to read that line as a fact. I was wrong. Verify it before giving it a column in the report.

## The biggest bot-hit total mostly measured noise

Our edge logs put the first two tiers side by side. The `/robots.txt` requests show the identity problem; the homepage shows why verification alone still leaves too much in the total.

![Server log requests separated into claimed hits, verified hits, and user-triggered page fetches](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/d722f6ec-a8de-49f6-80d3-f4eac3d4779a.png)

| Path          | Claimed hits | Verified hits | Verified share | User-triggered fetches                                                        |
| ------------- | -----------: | ------------: | -------------: | ----------------------------------------------------------------------------- |
| `/robots.txt` |       15,093 |         3,972 |            26% | 1,224 permission checks, almost all Claude-User; **not page-content fetches** |
| Homepage `/`  |      \~6,300 |         1,908 |            30% | 787 live-answer fetches, almost all ChatGPT-User                              |

**Three quarters of the claimed `/robots.txt` traffic failed the IP check.** The mechanism is familiar: a user-agent string is free text. In our logs, requests posing as ChatGPT-User also hunted `.env` and configuration paths rather than reading real pages. That is the sort of infrastructure-chasing behaviour Digimirror flags in fake bot traffic.

The homepage row needs a second cut. About 30% of its claimed fetches were verified, and 787 of those 1,908 verified fetches came from a live-answer fetcher. That is roughly 41% of the verified homepage total. The remaining verified requests should not be quietly promoted to answer activity just because the IP was real. They may belong to other kinds of crawling or repeat requests; this table does not assign each one a purpose.

A conventional bot-hits graph would make both paths look busy. Neither total tells a marketing owner which page might supply an answer. **Report the verified page fetch, not the weather around it.**

## The homepage gets checked; a dated post gets fetched

The homepage drew **1,908 verified fetches and 787 live-answer fetches**, almost all from ChatGPT-User, in our 30-day logs. I would expect a homepage to be consulted when an assistant needs to check what groas is or does. The count shows that the page was fetched; it does not tell me which prompt caused each request, what the assistant concluded, or whether a URL appeared in the response.

One dated changelog post told a more useful story. It drew **671 verified fetches and 417 live-answer fetches**, with both ChatGPT-User and Claude-User pulling the page body. About 62% of its verified fetches were in that user-triggered group. The post was also **97% verified** among its claimed bot requests. It states a particular change, when it happened, and what it costs. An assistant can reach those details in one page fetch rather than follow a list of links.

The hubs looked busy until I applied the same filter:

| Page type            | Verified fetches | Live-answer fetches | What the comparison supports                                          |
| -------------------- | ---------------: | ------------------: | --------------------------------------------------------------------- |
| Dated changelog post |              671 |                 417 | This page attracted substantial user-triggered fetching               |
| Category pages       |  Around 250 each |                   0 | Verified crawling did not translate into observed live-answer fetches |
| Blog index           |              204 |                   4 | The index was visited, but rarely by a live-answer fetcher            |

All figures in this table come from the same 30-day groas edge-log pull. **The 417 is a fetch count, not a count of confirmed citations or distinct answers.** Logs can show a user-triggered assistant requesting page content. They cannot show the text the user saw. That limit is worth keeping in the caption of any chart built from this data, not hiding in a footnote after the sales team has forwarded it.

Still, the page-level contrast is hard to ignore. A hub can help a crawler find its way around and finish with zero observed live-answer fetches. A specific, dated page can give a fetcher the detail it came for. Add the two into one brand-level bot total and the distinction disappears.

### Claude’s permission checks are not page reads

Claude-User accounted for almost all of the **1,224 user-triggered requests to `/robots.txt`** in our logs. A permission check can precede a page fetch, so counting both requests as content engagement makes one possible page visit look like more activity than it is. In this dataset, removing those checks leaves Claude’s observed content fetching concentrated on the changelog post and a handful of documentation pages.

**Exclude `/robots.txt` before ranking content.** It tells you a fetcher checked permission, not that it read an answer on that path. The distinction is especially important here because the robots count is larger than the live-answer fetch count for any individual content page in the table.

Use the right log source, too. [Behind Cloudflare, cached responses can miss origin logs](https://www.siteauditlint.com/blog/cloudflare-for-seo). If you rely only on origin logs, a request that the edge served may never appear in your analysis. Cloudflare Security Analytics or Logpush gives you the edge view I used for this comparison. First get the requests; then classify them.

Gemini is a limit of this method, not a zero in the results. As [Digimirror explains, Google exposes one Googlebot crawl identity and a Google-Extended permission flag](https://digimirror.io/your-bot-traffic-logs-are-lying-to-you/). From IP ranges alone, I cannot split Gemini training, retrieval, and user-triggered fetches the way I can examine distinct OpenAI or Anthropic fetchers. [Training and live retrieval serve different purposes](https://digimirror.io/your-bot-traffic-logs-are-lying-to-you/), but the relevant Gemini activity is not separately identifiable in this log analysis. Label that gap **unverifiable from these logs**, not absent.

## Pair page-fetch counts with what assistants actually say

A fetched page is a lead, not a confirmed citation. To check the other end of the chain, I would look at buyer prompts and the resulting answers alongside the edge logs. [Meev’s three-part check covers mentions, citations, and framing](https://meev.ai/articles/how-to-track-ai-brand-visibility-before-rivals-do): whether the answer names your brand, whether it cites your URL, and how it describes you. Those are different observations. Semrush data cited there found that [only 6% to 27% of brands mentioned in AI answers were also cited as the source](https://meev.ai/articles/how-to-track-ai-brand-visibility-before-rivals-do). A mention is not a citation; a fetch is not either.

For that answer-side check, take **50 to 100 real buyer prompts**, run each across ChatGPT, Perplexity, Gemini, Claude, and AI Overviews three times, and record the named brands, cited URLs, and framing. Keep the exact prompt and engine with each result. [Sona’s teardown of AEO checkers notes that visibility scores from different vendors are not comparable when their prompts, engines, and weighting differ](https://www.sona.com/blog/how-to-choose-an-aeo-checker-that-measures-more-than-mentions/). One blended percentage would erase the same distinctions the logs helped uncover.

Then look for a buyer question that appears in answers but has no clear page on your site to support it. The changelog result suggests what to try next: one page that addresses the question directly, puts its facts near the top, carries a date, and can be fetched as clean HTML. That is the same practical page work I describe in [how to make one page easier to cite](https://groas.com/post/how-to-get-cited-in-google-s-ai-overview). It is a candidate to test, not a promise that the next assistant answer will cite it.

![A dated answer page attracting user-triggered fetches while hub pages receive none](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/5fa60669-6307-4c17-9939-0e057c9761c1.png)

## Test your own pages without counting permission checks as wins

**Question:** Which pages attract verified, user-triggered content fetches, and which merely collect crawler traffic?

Use a prospective check rather than declaring victory when a bot chart rises:

1. **Set up the pull.** Collect 30 days of edge logs. If you are behind Cloudflare, use Security Analytics or Logpush rather than relying only on origin logs; otherwise use the server logs available to you. Keep the requested path, user-agent, source IP, and time together so you can inspect each request rather than just a daily total.
2. **Apply the identity control.** Filter requests claiming to be AI bots, then check source IPs against the relevant operator ranges for OpenAI, Anthropic, and Perplexity. Keep claimed and verified counts in separate columns. Do not let a convincing user-agent string stand in for verification.
3. **Apply the content control.** Separate user-triggered fetches of actual page URLs from training or other crawl activity where the bot identity permits that distinction. Exclude `/robots.txt` permission checks from the page-content ranking. Keep Gemini separately labelled where the log cannot establish its fetch purpose.
4. **Measure by page and engine.** Rank pages by verified live-answer fetches, with ChatGPT-User and Claude-User shown separately where possible. Compare that list with the claimed-hit and verified-hit lists. Repeat the pull next month to see whether a newly written answer page earns a row.

I would expect the three rankings to tell different stories. Our own robots and homepage verified shares, **26% and 30%**, are a reason to check your totals, not a benchmark I can promise for another site. Likewise, the **97% verified** share of our dated post describes that page, not a rule about dated posts everywhere. If your verified share is high across the site, check whether filtering at the edge changed what reached your logs before you call the traffic clean.

The result changes the next editorial decision either way. If a new, specific page draws verified user-triggered fetches, compare its prompts and cited URLs with what you intended it to answer. If it draws only routine verified crawls, do not present those as live-answer visibility. Recheck the page and the question it addresses before writing more of the same.

## Put the page-level number in next month’s report

I would replace the headline bot-hits total with **verified live-answer fetches per page, split by engine**. For this groas log window, the useful rows are the homepage’s **787** fetches, almost all ChatGPT-User; the changelog post’s **417** across ChatGPT-User and Claude-User; the category pages’ **zero**; and the blog index’s **four**. Keep claimed and verified traffic as diagnostic columns, not the result you lead with. Keep citations and answer wording in a separate prompt-level check.

Before celebrating any fall in traffic, check what fell. The [edge-log guidance on crawler access](https://www.siteauditlint.com/blog/cloudflare-for-seo) distinguishes search fetchers such as OAI-SearchBot and PerplexityBot from GPTBot training traffic. A firewall change that removes spoofed requests cleans up the claimed-hit total. A change that blocks the fetchers you need for answer discovery can remove the opportunity for a page to be fetched at all.

I use this distinction inside [groas](https://groas.com/) because the useful work is continuous: checking which buyer prompts name us, which pages assistants fetch, and which questions still lack a fetchable page. If you run the analysis yourself, keep the claim modest and the method strict. Verify the IP. Exclude permission checks. Rank actual pages by user-triggered fetches. Then check the answers before calling any fetch a citation.

Most AI-visibility measurement leads with bot hits. Our edge logs show why I would delete that line from the headline and write the next page for the question a real fetcher came to answer.

## Frequently Asked Questions

### Why are total bot hits a bad way to measure AI visibility?

Total bot hits are a poor metric because many requests claiming an AI-bot identity are not verified. In one log, 15,093 claimed fetches to robots.txt yielded only 3,972 verified requests, which was 26%.

### What are the three tiers used to classify AI-bot requests in the logs?

The three tiers are Claimed hits, Verified hits, and Live-answer fetches. Claimed hits are requests with an AI-bot name in the user-agent string, while Verified hits are claimed requests whose source IP matched the operator’s published range.

### Why is source IP verification important when tracking AI bot traffic?

The source IP is the first check because a user-agent string is merely a claim of identity. Without IP verification, the traffic total can include impersonators and routine crawlers that pose as known bot names.

### Which type of page is generally more likely to attract useful AI-assistant fetching?

A specific, dated changelog post is more likely to attract useful fetching than a hub like the homepage. For example, the changelog post drew 417 live-answer fetches and was 97% verified among claimed bot requests.

### Should I count requests to robots.txt as content engagement?

No, requests to /robots.txt should be excluded before ranking content. A permission check only tells you that a fetcher checked access, not that it read an answer on that specific content path.

### What should replace the headline total of bot hits in reports?

Reports should be based on verified live-answer fetches per page, split by engine. For example, the homepage drew 787 verified live-answer fetches, almost all from ChatGPT-User.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-4#webpage","url":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-4","name":"We Logged 30 Days of AI Bot Traffic. The Useful Number Wasn’t Bot Hits","description":"In groas’s 30-day edge logs, only 26% of claimed AI-bot fetches to robots.txt were verified. One dated post drew 417 user-triggered fetches. Here’s what to measure instead.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/5d0c50c0-b85e-492f-84f8-12085b7475db.png"},"breadcrumb":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-4#breadcrumb"},"dateModified":"2026-10-18T05:34:47.278Z","mainEntity":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-4#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-4#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"We Logged 30 Days of AI Bot Traffic. The Useful Number Wasn’t Bot Hits","item":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-4"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-4#article","url":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-4","mainEntityOfPage":{"@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-4#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"We Logged 30 Days of AI Bot Traffic. The Useful Number Wasn’t Bot Hits","description":"In groas’s 30-day edge logs, only 26% of claimed AI-bot fetches to robots.txt were verified. One dated post drew 417 user-triggered fetches. Here’s what to measure instead.","datePublished":"2026-10-18T05:34:47.199Z","dateModified":"2026-10-18T05:34:47.278Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/5d0c50c0-b85e-492f-84f8-12085b7475db.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"fetches, verified, page, bot, logs, requests, hits, robots txt","wordCount":2121,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own-4#faq","mainEntity":[{"@type":"Question","name":"Why are total bot hits a bad way to measure AI visibility?","acceptedAnswer":{"@type":"Answer","text":"Total bot hits are a poor metric because many requests claiming an AI-bot identity are not verified. In one log, 15,093 claimed fetches to robots.txt yielded only 3,972 verified requests, which was 26%."}},{"@type":"Question","name":"What are the three tiers used to classify AI-bot requests in the logs?","acceptedAnswer":{"@type":"Answer","text":"The three tiers are Claimed hits, Verified hits, and Live-answer fetches. Claimed hits are requests with an AI-bot name in the user-agent string, while Verified hits are claimed requests whose source IP matched the operator’s published range."}},{"@type":"Question","name":"Why is source IP verification important when tracking AI bot traffic?","acceptedAnswer":{"@type":"Answer","text":"The source IP is the first check because a user-agent string is merely a claim of identity. Without IP verification, the traffic total can include impersonators and routine crawlers that pose as known bot names."}},{"@type":"Question","name":"Which type of page is generally more likely to attract useful AI-assistant fetching?","acceptedAnswer":{"@type":"Answer","text":"A specific, dated changelog post is more likely to attract useful fetching than a hub like the homepage. For example, the changelog post drew 417 live-answer fetches and was 97% verified among claimed bot requests."}},{"@type":"Question","name":"Should I count requests to robots.txt as content engagement?","acceptedAnswer":{"@type":"Answer","text":"No, requests to /robots.txt should be excluded before ranking content. A permission check only tells you that a fetcher checked access, not that it read an answer on that specific content path."}},{"@type":"Question","name":"What should replace the headline total of bot hits in reports?","acceptedAnswer":{"@type":"Answer","text":"Reports should be based on verified live-answer fetches per page, split by engine. For example, the homepage drew 787 verified live-answer fetches, almost all from ChatGPT-User."}}]}]}
```