Exclusive finding · Findings · 2026-10-07 · By VSNARY | Emmanuel Orta · 10 views

Eight weeks of AI crawlers on one site: who was really who

Every request that claimed a named crawler on crawlcheck.io from 13 August to 7 October 2026, checked against the ranges its operator publishes. The index crawlers were nearly always real, the browsing agents far less so, three names only ever arrived forged, and every verified agent kept to robots.txt.

A request arrives with GPTBot in its user agent. Is it OpenAI? The string is a claim anyone can type; the only check that settles it is the source address against the ranges the operator publishes. This site has made that check on every request since 13 August 2026. This post reads the record on 7 October: one property, eight weeks, every named agent that actually came, counted one by one. It is a census of this site, not a sample of the web.

Which crawlers really visited? #

Seven index crawlers accounted for almost all verified traffic. For each, the large majority of claims came from the operator's own ranges, so the name is a reliable signal provided the address is checked.

Which names are forged most? #

The pattern held for the whole window: forgery is rare for crawlers that index, and much more common for agents that fetch a page because a person asked in a chat. Those are the names a scraper picks when it wants to look like someone's assistant, because sites that block GPTBot often let ChatGPT-User through. For Perplexity-User, forged claims outnumbered real ones. If your access policy treats user-triggered agents more generously than crawlers, that policy is being used, and the address check is the only thing standing between it and anyone who read the docs. Per-crawler rates move week to week; the current ones are below.

Forged crawler identities #

13.5% of the requests that named themselves as a known crawler here were not that crawler. 20392 of 150680 checkable claims came from an address outside the range the operator publishes.

A user-agent is a claim, not an identity. Verified means the source IP sits inside a range the operator publishes — OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish one. Unverifiable is not forgery: some operators publish no range at all, so their requests can be neither confirmed nor accused, and they are excluded from the rate rather than counted against it. This is traffic to this site only, and this site is small — it is a floor on the problem, not a survey of the web.

Three names that only ever arrive as forgeries #

Google-Extended and anthropic-ai are names a site writes in robots.txt to control training use. The operator's crawler identifies as Googlebot or ClaudeBot and reads the token in the file, so a request carrying the token was written by someone who learned the names from a robots.txt guide. It is a forgery by construction and cheap to drop at the edge (more in a name Google says it never sends).

Did the verified agents follow robots.txt? #

Yes, on this site. No verified request fetched a path this site's robots.txt disallows, including paths disallowed in a group that names the agent specifically. The busiest minute of any verified agent belonged to Googlebot, at 3 requests. The crawlers people worry about, when they are really themselves, read the rules and kept to them here for eight weeks. That says nothing about anyone else's rules or what they did with the pages; it is one property and one robots.txt, but it is a measured baseline where there was none.

The visits nobody could verify #

In the window, 47,718 visits used 17 names that had no published address range to check against. The real agent and an impersonator look identical, so none of these can be attributed to the operator. That is not an accusation; the traffic may be entirely genuine.

Amazonbot alone made more requests here than Googlebot and GPTBot together. When this record was read, none of them could be told from anyone who typed the name; Amazon has since published ranges, so the same traffic is now checkable. That is the whole argument in one row: a site cannot allow a crawler it cannot recognise without allowing everyone who imitates it, and publishing a range file is what lets a crawler be let in by name.

What the first day on our smaller sites showed #

Before this record, we counted visits on a handful of small business sites we operate. Bytespider made the most visits of any named agent, and the path it requested most was a PDF in an uploads folder, not the homepage. As first published, we called those visits verified. That was wrong: ByteDance publishes no ranges, so they are unverifiable, and a name with no published ranges is now always recorded that way. The same day, one agent name arrived both from the operator's ranges and from a cloud network; the cloud requests were our own testing, which is exactly why unverified is not proof of an impostor.

Run the same check on your own logs #

• Check each named request's address against the list for that exact agent, refreshed daily. The method and files are in how to verify crawlers by IP.
• Report three columns: verified, forged, unverifiable. Divide the forgery rate by verified plus forged only; dividing by everything counts “no feed published” as “fine”.
• Keep the Host header in your log format. It shows whether a crawler prefers your apex or your www, which affects which copy of a page gets indexed, and the default format throws it away.
• Drop requests carrying robots.txt tokens such as Google-Extended as user agents.

Method #

Every request to crawlcheck.io whose user agent names a known agent was checked at arrival against the address ranges that agent's operator publishes. Verified means inside the ranges; forged means the name with an address outside every range the operator publishes for it. Agents whose operators published no ranges were counted but never judged. Robots.txt conduct counts verified fetches of disallowed paths, so it is a lower bound. Counts are as of 7 October 2026; the live figures above change daily. Which operators publish what is listed per agent on the AI crawler list.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Are Googlebot and GPTBot requests usually real?

On crawlcheck.io over eight weeks, the large majority of requests naming the big index crawlers came from their operators' published ranges. The name is a reliable signal for those crawlers, but only once the address has been checked.

Which AI agent names are forged most often?

The user-triggered fetchers, such as ChatGPT-User, Claude-User and Perplexity-User. Scrapers pick them because sites often let a person's assistant through when they block crawlers. For Perplexity-User, forged claims outnumbered real ones on this site.

Why is every request named Google-Extended a forgery?

Google-Extended is a robots.txt token, not a crawler. Google's crawlers identify as Googlebot and read the token in the file, so no genuine request carries it as a user agent.

Did verified AI crawlers respect robots.txt?

On this site, yes. Across eight weeks no verified request fetched a path the site's robots.txt disallows, including paths disallowed for that agent by name. That is one site's record, not a verdict on every site.

Can Amazonbot be verified?

Yes, now. When this record was read Amazon published no ranges, so every Amazonbot request was unverifiable. Amazon has since published a list, so the address can be checked against it.

Why should a forgery rate exclude unverifiable requests?

Because nothing can be concluded about them. Counting them as genuine flatters the rate; counting them as forged accuses real traffic. Divide by verified plus forged and report the unverifiable separately.

Related findings

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works