Findings · 2026-08-15 · By VSNARY | Emmanuel Orta · 0 views

How much AI crawler traffic is forged? Checked against IP ranges

A user-agent string is a claim, not an identity. Every request here that claims a named crawler is checked against the IP ranges that operator publishes — and a measurable share of claims fail the check.

A request arrives saying User-Agent: GPTBot. That is not an identity; it is a claim, typed by whoever sent the request. Most crawler statistics on the web repeat the claim. This page adjudicates it: OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity publish the IP ranges their crawlers send from, so a claim can be checked against the operator’s own list — and a measurable share of claims fail.

Three outcomes, never collapsed into two

Every request claiming a named crawler lands in exactly one bucket, and keeping the third one separate is the method:

• Verified — the source IP falls inside the range the operator publishes.
• Spoofed — the operator publishes ranges and this IP is outside all of them. Someone else is wearing the name.
• Unverifiable — the operator publishes no range feed. This is not forgery, and counting it as forgery would inflate the rate. Amazonbot and Meta-ExternalAgent live here: their requests can be neither confirmed nor accused.

The forgery rate below divides spoofed claims by adjudicable claims only — claims where a published feed made a verdict possible. Dividing by every claim would understate it by treating “no feed published” as “fine”.

That third bucket is not a rounding error. Of every request here that named a known crawler, well over a third named an operator that publishes nothing to check against — Meta’s agent is the single largest claimed identity in the whole record, and not one of its requests can be confirmed or denied. A tool that reports a forgery rate without telling you how much of its traffic was unadjudicable has hidden its own uncertainty inside a percentage. Which operators publish a feed and which do not is its own measurement, and it changes.

The rate, read from the live record

Forged crawler identities #

7.1% of the requests that named themselves as a known crawler here were not that crawler. 1956 of 27659 checkable claims came from an address outside the range the operator publishes.

A user-agent is a claim, not an identity. Verified means the source IP sits inside a range the operator publishes — OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish one. Unverifiable is not forgery: some operators publish no range at all, so their requests can be neither confirmed nor accused, and they are excluded from the rate rather than counted against it. This is traffic to this site only, and this site is small — it is a floor on the problem, not a survey of the web.

The floor under every per-crawler number

A rate needs a denominator big enough to be a pattern. So no overall rate is published, and no individual crawler is named or charted, until there are enough claims behind it to mean something. One spoofed request out of one is a 100% forgery rate, and it is evidence of nothing.

The per-crawler floor is newer, and it was added because the chart did not have it. The citation sentence already refused to name a most-forged crawler on thin evidence; the bar chart above it applied no minimum, so its top rows — the ones a reader takes as the headline — could be a crawler seen twice and spoofed twice. Two names were sitting there at 100% on two claims each. The picture was saying something the sentence underneath it explicitly refused to say. Same floor in both places now.

What survives that floor is more interesting than what it removed. Two names still read 100% with real volume behind them — dozens of adjudicable claims each, every one from outside the ranges their operator publishes. Both are opt-out signalling agents rather than fetchers, which is precisely the kind of name nobody would bother to forge unless it opened something.

Why anyone forges a crawler name

Because the name opens doors. Sites allowlist Googlebot in rate limiters and paywalls; increasingly they allowlist AI crawlers they want to be cited by. A scraper wearing GPTBot’s name inherits GPTBot’s welcome — and GPTBot’s blame, since the operator whose logs fill with the name draws conclusions about OpenAI, not about the impersonator. The most-imitated names are the most valuable ones, which is why the flagship crawlers top the forged list rather than obscure ones.

The pattern in the table supports that directly. The names with the most claims are forged least — the crawler with the largest published range feed and the most traffic sits in single-digit percentages, because forging it is the easiest thing in the world to catch. The interactive user-triggered agents, the ones a site is most likely to welcome and least likely to check, run several times higher. Forgery concentrates where the check is unlikely to be run.

Group by address before you conclude anything

The rate is a property of claims, not of machines, and the two come apart badly. We nearly published a 57% crawler block rate from our own logs before grouping the refusals by source address: all of them came from one IP presenting seven different crawler identities in a single day. One host cannot be seven companies. Counting those as blocked crawler traffic did not merely inflate a number — it inverted the conclusion, because the genuine crawlers in the same logs were refused zero times.

So before you conclude anything from a log full of crawler names: group refusals and claims by source IP, count how many distinct identities each address claims, and check what paths they asked for. Any single address wearing more than one named crawler is forging at least one of them.

What to do with this if you run a site

Never make an access decision on a user-agent string alone. Every operator in the verified column publishes its ranges precisely so you can check; a robots rule keys on the claimed name, but a firewall allowlist or a log analysis should key on the verified identity. If your analytics say an AI crawler is hammering you, the first question is whether it was actually them.

The verification logic here is open source and dependency-free, so you can run it against your own logs rather than taking this page’s word for it: it fetches each operator’s published feed, parses the ranges, and returns verified, spoofed or unverifiable per line — with unverifiable kept separate, for the same reason it is kept separate here. Every agent named on this page, who sends it and what it is for, is listed on the crawler reference.

What this sample is, stated plainly

These are requests to this instrument’s own endpoints and to sites it watches — first-party observation, no sampling, no third-party logs. It is not a random sample of the web, and the rate here should be quoted as “measured by CrawlCheck on its own traffic”, not as a web-wide figure. The numbers are recomputed from the live record when this page loads; a copy of the number ages, the URL does not.

One further limit, because it cuts the other way. A verified request proves the address belongs to the operator, not that the operator behaved well; and a spoofed one proves the address is outside the published range, not who sent it. Ranges also change, and a feed read an hour after an operator adds a block will call the new addresses spoofed. That is a false positive with a known cause, it resolves itself on the next read, and it is the reason no single request here is ever published as an accusation against a named company — only rates, over floors, with the unadjudicable share stated beside them.

Where the verification comes from #

Ten published feeds, all one shape: OpenAI's for GPTBot, its search bot and its user-triggered fetcher; Anthropic's; Perplexity's two; Google's common crawlers and user-triggered fetchers; Bing's; Apple's. The feeds are cached for a day and a failed refresh keeps the previous copy, so a transient outage at an operator can never reclassify real crawlers as unverified. An address inside a feed is verified. An address outside every feed for an operator that publishes one is spoofed. An agent whose operator publishes nothing is unverifiable, and unverifiable is not forgery. The crawler list links each feed.

Why the headline number is verified only #

Every count this site publishes about crawler traffic is the verified count. The claimed count is shown beside it, never instead of it, after we found our own testing filed under an AI company's name. A dataset whose largest contributor is its own author is not a dataset. The glossary defines verified crawler, forged crawler identity and forward-confirmed reverse DNS, the older check that the address feeds replace.

Questions this post answers

How can you tell if AI crawler traffic is forged?

Check the source address against the IP ranges the operator publishes. The user-agent is a claim; only the published range makes that claim checkable.

Why three outcomes instead of forged or genuine?

Because some operators publish no range at all. Those requests can be neither confirmed nor accused, and folding them into either bucket produces a number that is wrong by construction.

What should a site owner do with a forgery rate?

Treat it as a reason to verify by address before blocking by name. Blocking a user-agent string stops the honest traffic and none of the forged traffic.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works