We Logged 30 Days of AI Bot Traffic. The Useful Number Wasn’t Bot Hits

Something claiming to be an AI bot asked methodlantern.com for /robots.txt 15,093 times in 30 days. Our edge logs show that only 3,972 of those requests, or 26%, came from IPs in the operators’ published ranges. The rest could make the claim in a user-agent string; they could not back it up.

That gap makes total bot hits a poor AI-visibility metric. A request to check crawl permissions is not a request for an answer page, either. To find the useful signal, I split the logs into claimed hits, verified hits, and verified, user-triggered fetches of actual pages. The last group gets closest to showing which pages an assistant might use in a live response. It still does not prove what the assistant ultimately said or cited.

The dataset separates a claim from a page fetch

I pulled 30 days of edge logs for methodlantern.com and sorted requests claiming an AI-bot identity into three tiers. The distinction matters because the same dashboard total can contain impersonators, routine crawlers, permission checks, and fetches prompted by a person asking an assistant a question.

A user-agent is a claim; the source IP is the first check. Digimirror describes the same claim-versus-proof distinction. In one new-domain audit, it found that only 6 of 33 requests claiming to be live assistant fetches were real; only 107 of 799 Googlebot-labeled requests came from verified Google IPs. Promptwatch counted about 18% less AI-crawler traffic on average after IP-range verification. And CHEQ puts false representation at 1 in 7 invalid retail visits, describing scrapers that wear known crawler names while real agents can look like ordinary Chrome traffic.

Those are different datasets with different denominators, not a pooled industry rate. They do explain why I stopped accepting a bot name in a log as evidence of AI interest. I used to read that line as a fact. I was wrong. Verify it before giving it a column in the report.

The biggest bot-hit total mostly measured noise

Our edge logs put the first two tiers side by side. The /robots.txt requests show the identity problem; the homepage shows why verification alone still leaves too much in the total.

Three quarters of the claimed /robots.txt traffic failed the IP check. The mechanism is familiar: a user-agent string is free text. In our logs, requests posing as ChatGPT-User also hunted .env and configuration paths rather than reading real pages. That is the sort of infrastructure-chasing behaviour Digimirror flags in fake bot traffic.

The homepage row needs a second cut. About 30% of its claimed fetches were verified, and 787 of those 1,908 verified fetches came from a live-answer fetcher. That is roughly 41% of the verified homepage total. The remaining verified requests should not be quietly promoted to answer activity just because the IP was real. They may belong to other kinds of crawling or repeat requests; this table does not assign each one a purpose.

A conventional bot-hits graph would make both paths look busy. Neither total tells a marketing owner which page might supply an answer. Report the verified page fetch, not the weather around it.

The homepage gets checked; a dated post gets fetched

The homepage drew 1,908 verified fetches and 787 live-answer fetches, almost all from ChatGPT-User, in our 30-day logs. I would expect a homepage to be consulted when an assistant needs to check what groas is or does. The count shows that the page was fetched; it does not tell me which prompt caused each request, what the assistant concluded, or whether a URL appeared in the response.

One dated changelog post told a more useful story. It drew 671 verified fetches and 417 live-answer fetches, with both ChatGPT-User and Claude-User pulling the page body. About 62% of its verified fetches were in that user-triggered group. The post was also 97% verified among its claimed bot requests. It states a particular change, when it happened, and what it costs. An assistant can reach those details in one page fetch rather than follow a list of links.

The hubs looked busy until I applied the same filter:

All figures in this table come from the same 30-day groas edge-log pull. The 417 is a fetch count, not a count of confirmed citations or distinct answers. Logs can show a user-triggered assistant requesting page content. They cannot show the text the user saw. That limit is worth keeping in the caption of any chart built from this data, not hiding in a footnote after the sales team has forwarded it.

Still, the page-level contrast is hard to ignore. A hub can help a crawler find its way around and finish with zero observed live-answer fetches. A specific, dated page can give a fetcher the detail it came for. Add the two into one brand-level bot total and the distinction disappears.

Claude’s permission checks are not page reads

Claude-User accounted for almost all of the 1,224 user-triggered requests to /robots.txt in our logs. A permission check can precede a page fetch, so counting both requests as content engagement makes one possible page visit look like more activity than it is. In this dataset, removing those checks leaves Claude’s observed content fetching concentrated on the changelog post and a handful of documentation pages.

Exclude /robots.txt before ranking content. It tells you a fetcher checked permission, not that it read an answer on that path. The distinction is especially important here because the robots count is larger than the live-answer fetch count for any individual content page in the table.

Use the right log source, too. Behind Cloudflare, cached responses can miss origin logs. If you rely only on origin logs, a request that the edge served may never appear in your analysis. Cloudflare Security Analytics or Logpush gives you the edge view I used for this comparison. First get the requests; then classify them.

Gemini is a limit of this method, not a zero in the results. As Digimirror explains, Google exposes one Googlebot crawl identity and a Google-Extended permission flag. From IP ranges alone, I cannot split Gemini training, retrieval, and user-triggered fetches the way I can examine distinct OpenAI or Anthropic fetchers. Training and live retrieval serve different purposes, but the relevant Gemini activity is not separately identifiable in this log analysis. Label that gap unverifiable from these logs, not absent.

Pair page-fetch counts with what assistants actually say

A fetched page is a lead, not a confirmed citation. To check the other end of the chain, I would look at buyer prompts and the resulting answers alongside the edge logs. Meev’s three-part check covers mentions, citations, and framing: whether the answer names your brand, whether it cites your URL, and how it describes you. Those are different observations. Semrush data cited there found that only 6% to 27% of brands mentioned in AI answers were also cited as the source. A mention is not a citation; a fetch is not either.

For that answer-side check, take 50 to 100 real buyer prompts, run each across ChatGPT, Perplexity, Gemini, Claude, and AI Overviews three times, and record the named brands, cited URLs, and framing. Keep the exact prompt and engine with each result. Sona’s teardown of AEO checkers notes that visibility scores from different vendors are not comparable when their prompts, engines, and weighting differ. One blended percentage would erase the same distinctions the logs helped uncover.

Then look for a buyer question that appears in answers but has no clear page on your site to support it. The changelog result suggests what to try next: one page that addresses the question directly, puts its facts near the top, carries a date, and can be fetched as clean HTML. That is the same practical page work I describe in how to make one page easier to cite. It is a candidate to test, not a promise that the next assistant answer will cite it.

Test your own pages without counting permission checks as wins

Question: Which pages attract verified, user-triggered content fetches, and which merely collect crawler traffic?

Use a prospective check rather than declaring victory when a bot chart rises:

1. Set up the pull. Collect 30 days of edge logs. If you are behind Cloudflare, use Security Analytics or Logpush rather than relying only on origin logs; otherwise use the server logs available to you. Keep the requested path, user-agent, source IP, and time together so you can inspect each request rather than just a daily total.
2. Apply the identity control. Filter requests claiming to be AI bots, then check source IPs against the relevant operator ranges for OpenAI, Anthropic, and Perplexity. Keep claimed and verified counts in separate columns. Do not let a convincing user-agent string stand in for verification.
3. Apply the content control. Separate user-triggered fetches of actual page URLs from training or other crawl activity where the bot identity permits that distinction. Exclude /robots.txt permission checks from the page-content ranking. Keep Gemini separately labelled where the log cannot establish its fetch purpose.
4. Measure by page and engine. Rank pages by verified live-answer fetches, with ChatGPT-User and Claude-User shown separately where possible. Compare that list with the claimed-hit and verified-hit lists. Repeat the pull next month to see whether a newly written answer page earns a row.

I would expect the three rankings to tell different stories. Our own robots and homepage verified shares, 26% and 30%, are a reason to check your totals, not a benchmark I can promise for another site. Likewise, the 97% verified share of our dated post describes that page, not a rule about dated posts everywhere. If your verified share is high across the site, check whether filtering at the edge changed what reached your logs before you call the traffic clean.

The result changes the next editorial decision either way. If a new, specific page draws verified user-triggered fetches, compare its prompts and cited URLs with what you intended it to answer. If it draws only routine verified crawls, do not present those as live-answer visibility. Recheck the page and the question it addresses before writing more of the same.

Put the page-level number in next month’s report

I would replace the headline bot-hits total with verified live-answer fetches per page, split by engine. For this groas log window, the useful rows are the homepage’s 787 fetches, almost all ChatGPT-User; the changelog post’s 417 across ChatGPT-User and Claude-User; the category pages’ zero; and the blog index’s four. Keep claimed and verified traffic as diagnostic columns, not the result you lead with. Keep citations and answer wording in a separate prompt-level check.

Before celebrating any fall in traffic, check what fell. The edge-log guidance on crawler access distinguishes search fetchers such as OAI-SearchBot and PerplexityBot from GPTBot training traffic. A firewall change that removes spoofed requests cleans up the claimed-hit total. A change that blocks the fetchers you need for answer discovery can remove the opportunity for a page to be fetched at all.

I use this distinction inside groas because the useful work is continuous: checking which buyer prompts name us, which pages assistants fetch, and which questions still lack a fetchable page. If you run the analysis yourself, keep the claim modest and the method strict. Verify the IP. Exclude permission checks. Rank actual pages by user-triggered fetches. Then check the answers before calling any fetch a citation.

Most AI-visibility measurement leads with bot hits. Our edge logs show why I would delete that line from the headline and write the next page for the question a real fetcher came to answer.