CrawlCheck

Findings · 2026-08-18 · By

Which AI crawlers can't be verified by IP range?

On 18 August, 251 of 950 requests claiming a named crawler came from operators who published no address range. Those requests are not forged; they are unfalsifiable. Two of those operators have since published ranges. Here is who is checkable and what to do about the rest.

A crawler claim can only be checked if its operator publishes something to check it against: a JSON IP feed, a reverse-DNS convention, or equivalent documentation. OpenAI, Anthropic, Google, Microsoft, Perplexity, Apple and now Amazon and Ahrefs do. Meta and several others publish nothing, so anything that can set a header can claim their names. Those requests are not forgeries; they are unfalsifiable. Report them as their own class, keep them out of forgery rates, and control them with rules that do not depend on identity.

When an operator publishes the address ranges its crawler uses, every request in its name becomes checkable: the address either sits inside the range or it does not. When it publishes nothing, a request wearing its name can be neither confirmed nor accused. Counting it as forged would be an accusation nobody can support; counting it as genuine would be a guess.

How much crawler traffic is unverifiable? #

Measured on this site on 18 August 2026: of 950 requests that named a known crawler, 251 could not be checked at all, because the operator published no range we could find.

Claimed identityClaims on 18 AugCheckable thenSince
Meta-ExternalAgent148No published rangeStill none
Amazonbot56No published rangeAmazon now publishes ranges
AhrefsBot31No published rangeAhrefs now publishes ranges
Bytespider7No published rangeStill none
Baiduspider4No published rangeSee the crawler list
DuckDuckBot2No published rangeSee the crawler list
CCBot2No published rangeStill none
Diffbot1No published rangeSee the crawler list

Meta-ExternalAgent alone was 148 of the 251, one of the most active crawlers reaching this site, with no published way to tell its requests from anyone typing its name. It has remained among the largest claimed identities in the longer record, and still the least checkable. Amazon and Ahrefs show the other direction: once an operator publishes a feed, its traffic becomes checkable, retroactively, for anyone who kept the addresses. The current status of every agent is on the AI crawler list.

The wider gap is in the catalogue itself. Most named crawler identities on record have no feed, and many have no operator anyone can name. An identity with no operator is not necessarily illegitimate, but there is no one to ask for a range, report abuse to, or hold to a stated policy.

19

Why unverifiable is worse than forged #

A forgery rate is a number that can be measured and driven down. An operator that publishes nothing creates a permanent gap: every site deciding whether to allow that crawler decides on a string an attacker can copy in one line. It also corrupts the statistics people publish. Work the same day's traffic three ways and you get three very different “forgery rates”, which is why crawler figures from different sources disagree so sharply.

Same traffic, three denominatorsverifiedforgedunverifiableTrust every nameunverifiable counted as genuine: rate too lowDistrust every nameunverifiable counted as forged: rate too highAdjudicateforged / (verified + forged); unverifiable reported apart
Each reading is published somewhere as “the AI crawler forgery rate”. Only the third measures forgery.

The same rule applies to any measurement: a check ends true, false or unverifiable, and the third is never quietly filed as the second. We once published a failing grade that came entirely from collapsing those into two: a site's firewall refused our address, and the report scored the refusal as though it were the site. The live forgery rate, with unverifiable claims kept apart, is below.

Citing this figure. The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged#the-rate. Sentence form, as of 2026-11-22: 11.6% of requests claiming a verifiable AI-crawler identity, measured by CrawlCheck on its own traffic (25509 of 220734 adjudicable claims), came from outside the operator's published IP ranges; the most-forged name was Google-Extended at 100%.

What does “publishes a range” actually mean? #

It is not one yes-or-no. Four shapes count as publication, and only some can be used without a human in the loop.

ShapeWhat it isUsable by a machine?
JSON feed at a documented URLA stable address returning the current CIDR listYes: fetch, parse, match
HTML page listing rangesA documentation page a person is expected to readOnly by scraping a layout that can change without notice
Reverse-DNS conventionA documented hostname pattern to resolve and confirmYes, with a forward-confirm lookup per new address
ASN“Our traffic comes from this network”Weakly: an ASN is far larger than a crawler fleet

One trap from checking the catalogue entry by entry: a name can look covered because a sibling name is. Several operators publish one feed for a family of agents, and at least one publishes feeds for two of its three named crawlers and none for the third. Record coverage per agent, against the operator's own page, not by whether the company is known to publish anything.

Ranges move: refresh, don't copy #

An allowlist compiled from a feed six months ago lists addresses that were right six months ago. The failure is quiet in both directions: real crawler traffic from new addresses starts getting blocked, while released addresses stay trusted. If you key anything on identity, re-read the feed on a schedule and rebuild the list from it. The method is in how to verify crawlers by IP.

What can I still enforce against an unverifiable crawler? #

  • Not identity rules. A rate limit or allow rule keyed on a name is lifted by changing the name, and cost or abuse attributed to a name is attributed to whoever typed it.
  • Everything else. Path rules, per-address rate limits, size limits and challenges work regardless of what the request calls itself. That is the layer worth investing in.
  • robots.txt, as a request. A rule aimed at an unverifiable agent is still worth writing, because well-behaved operators honour it, but it is politeness, not control. For operators that publish ranges, a firewall rule on the verified identity is enforceable; see how to block AI crawlers.

What would close the gap #

No standards body is needed. An operator that wants its crawler distinguishable from impersonators needs a stable URL returning its current ranges as JSON, a documented reverse-DNS convention as a fallback, and a page naming every token it sends. Publishing costs a small maintenance burden and buys the ability to be believed. Publishing nothing costs the operator nothing and hands the whole cost to every site it visits, each of which must choose between over-counting it and accusing it. That is a choice, not an absence.

Counts are traffic to this site only, read on 18 August 2026. This is a small site: the figures are a floor on the problem, not a survey of the web. Unverifiable means the operator published no range we could find; when that changes, the classification changes with it.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Which AI crawlers publish IP ranges?

OpenAI, Anthropic, Perplexity, Google, Microsoft and Apple publish address feeds, and Amazon and Ahrefs now do too. Meta-ExternalAgent, Bytespider and CCBot publish none, so a request claiming those names cannot be checked.

Why is an unverifiable claim worse than a forgery rate?

A forgery rate can be measured and acted on. An unfalsifiable claim cannot be confirmed or denied by anyone, so no amount of care on your side settles it.

How do I verify an AI crawler is real?

Check the request's source address against what the operator publishes: an IP feed at a documented URL, or a reverse-DNS hostname you resolve and then forward-confirm. If the operator publishes neither, there is nothing to check against and the claim cannot be adjudicated either way.

What does it mean when a crawler cannot be verified?

The operator publishes no address range, no reverse-DNS convention and no equivalent documentation, so a request with that name could be the operator or anyone else. It is not evidence of forgery; it is the absence of evidence in either direction.

Should I block crawlers I cannot verify?

Blocking by name does not work on an identity you cannot verify, because the sender chooses the name. Use controls that do not depend on who is asking: per-address rate limits, path rules, size limits and challenges. Keep identity rules for operators that publish ranges.

What does publishing a range cost an operator?

A JSON file at a stable URL and the discipline to keep it current. Leaving it out is a choice, not a technical limit.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works