CrawlCheck

Agent view · Findings · 2026-08-18

Which AI crawlers can't be verified by IP range?

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/the-crawlers-nobody-can-check: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/the-crawlers-nobody-can-check>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes16,720
words2,454
sha-25681efed9b37011f9639a44dbbf755c7e6fc86c3d431b26ee3837408bbcd1af0b5
repr-digestsha-256=:ge/tmzcBH5Y5pE2791XH5vyGw9Qxsm7jg3QIu80a8LU=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311500;expires=1795315100;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="kz5LZOQLS1gMYtpmE6fb4g";tag="crawlcheck-page"
signaturesig1=:/dqM/vlisZq/1XsioC8D1a5kTElH+YEX10YtaJjkHkTHXLI12Lvp42Mc/3MncaxJ0kN4XZv+k7ziFs6Q5C16Aw==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Findings · 2026-08-18 · By [VSNARY | Emmanuel Orta](/author)

# Which AI crawlers can't be verified by IP range?

On 18 August, 251 of 950 requests claiming a named crawler came from operators who published no address range. Those requests are not forged; they are unfalsifiable. Two of those operators have since published ranges. Here is who is checkable and what to do about the rest.

A crawler claim can only be checked if its operator publishes something to check it against: a JSON IP feed, a reverse-DNS convention, or equivalent documentation. OpenAI, Anthropic, Google, Microsoft, Perplexity, Apple and now Amazon and Ahrefs do. Meta and several others publish nothing, so anything that can set a header can claim their names. Those requests are not forgeries; they are unfalsifiable. Report them as their own class, keep them out of forgery rates, and control them with rules that do not depend on identity.

On this page

- How much crawler traffic is unverifiable?
- Why unverifiable is worse than forged
- What does “publishes a range” actually mean?
- Ranges move: refresh, don't copy
- What can I still enforce against an unverifiable crawler?
- What would close the gap

When an operator publishes the address ranges its crawler uses, every request in its name becomes checkable: the address either sits inside the range or it does not. When it publishes nothing, a request wearing its name can be neither confirmed nor accused. Counting it as forged would be an accusation nobody can support; counting it as genuine would be a guess.

## How much crawler traffic is unverifiable? #

Measured on this site on 18 August 2026: of 950 requests that named a known crawler, **251 could not be checked at all**, because the operator published no range we could find.

| Claimed identity | Claims on 18 Aug | Checkable then | Since |
| --- | --- | --- | --- |
| Meta-ExternalAgent | 148 | No published range | Still none |
| Amazonbot | 56 | No published range | Amazon now publishes ranges |
| AhrefsBot | 31 | No published range | Ahrefs now publishes ranges |
| [Bytespider](/glossary/bytespider) | 7 | No published range | Still none |
| Baiduspider | 4 | No published range | See the crawler list |
| DuckDuckBot | 2 | No published range | See the crawler list |
| [CCBot](/glossary/ccbot) | 2 | No published range | Still none |
| Diffbot | 1 | No published range | See the crawler list |

Meta-ExternalAgent alone was 148 of the 251, one of the most active crawlers reaching this site, with no published way to tell its requests from anyone typing its name. It has remained among the largest claimed identities in the longer record, and still the least checkable. Amazon and Ahrefs show the other direction: once an operator publishes a feed, its traffic becomes checkable, retroactively, for anyone who kept the addresses. The current status of every agent is on the [AI crawler list](/ai-crawlers).

The wider gap is in the catalogue itself. Most named crawler identities on record have no feed, and many have no operator anyone can name. An identity with no operator is not necessarily illegitimate, but there is no one to ask for a range, report abuse to, or hold to a stated policy.
19

## Why unverifiable is worse than forged #

A forgery rate is a number that can be measured and driven down. An operator that publishes nothing creates a permanent gap: every site deciding whether to allow that crawler decides on a string an attacker can copy in one line. It also corrupts the statistics people publish. Work the same day's traffic three ways and you get three very different “forgery rates”, which is why crawler figures from different sources disagree so sharply.
Each reading is published somewhere as “the AI crawler forgery rate”. Only the third measures forgery.
The same rule applies to any measurement: a check ends true, false or [unverifiable](/glossary/unverifiable), and the third is never quietly filed as the second. [We once published a failing grade](/blog/the-site-that-scored-15-because-it-refused-us) that came entirely from collapsing those into two: a site's firewall refused our address, and the report scored the [refusal](/glossary/refusal) as though it were the site. The live forgery rate, with unverifiable claims kept apart, is below.

**Citing this figure.** The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: `https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged#the-rate`. Sentence form, as of 2026-11-22: *11.5% of requests claiming a verifiable AI-crawler identity, measured by CrawlCheck on its own traffic (25509 of 221127 adjudicable claims), came from outside the operator's published IP ranges; the most-forged name was [Google-Extended](/glossary/google-extended) at 100%.*

## What does “publishes a range” actually mean? #

It is not one yes-or-no. Four shapes count as publication, and only some can be used without a human in the loop.

| Shape | What it is | Usable by a machine? |
| --- | --- | --- |
| JSON feed at a documented [URL](/glossary/url) | A stable address returning the current CIDR list | Yes: fetch, parse, match |
| HTML page listing ranges | A documentation page a person is expected to read | Only by scraping a layout that can change without notice |
| Reverse-DNS convention | A documented hostname pattern to resolve and confirm | Yes, with a forward-confirm lookup per new address |
| ASN | “Our traffic comes from this network” | Weakly: an ASN is far larger than a crawler fleet |

One trap from checking the catalogue entry by entry: a name can look covered because a sibling name is. Several operators publish one feed for a family of agents, and at least one publishes feeds for two of its three named crawlers and none for the third. Record coverage per agent, against the operator's own page, not by whether the company is known to publish anything.

## Ranges move: refresh, don't copy #

An allowlist compiled from a feed six months ago lists addresses that were right six months ago. The failure is quiet in both directions: real crawler traffic from new addresses starts getting blocked, while released addresses stay trusted. If you key anything on identity, re-read the feed on a schedule and rebuild the list from it. The method is in [how to verify crawlers by IP](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers).

## What can I still enforce against an unverifiable crawler? #

- **Not identity rules.** A [rate limit](/glossary/rate-limit) or allow rule keyed on a name is lifted by changing the name, and cost or abuse attributed to a name is attributed to whoever typed it.
- **Everything else.** Path rules, per-address rate limits, size limits and challenges work regardless of what the request calls itself. That is the layer worth investing in.
- **[robots.txt](/glossary/robots-txt), as a request.** A rule aimed at an unverifiable agent is still worth writing, because well-behaved operators honour it, but it is politeness, not control. For operators that publish ranges, a firewall rule on the verified identity is enforceable; see [how to block AI crawlers](/guides/how-to-block-ai-crawlers).

## What would close the gap #

No standards body is needed. An operator that wants its crawler distinguishable from impersonators needs a stable URL returning its current ranges as JSON, a documented reverse-DNS convention as a fallback, and a page naming every [token](/glossary/token) it sends. Publishing costs a small maintenance burden and buys the ability to be believed. Publishing nothing costs the operator nothing and hands the whole cost to every site it visits, each of which must choose between over-counting it and accusing it. That is a choice, not an absence.

Counts are traffic to this site only, read on 18 August 2026. This is a small site: the figures are a floor on the problem, not a survey of the web. Unverifiable means the operator published no range we could find; when that changes, the classification changes with it.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fthe-crawlers-nobody-can-check%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fthe-crawlers-nobody-can-check%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Which%20AI%20crawlers%20can)[Email](mailto:?subject=Which%20AI%20crawlers%20can)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Which AI crawlers publish IP ranges?

OpenAI, Anthropic, Perplexity, Google, Microsoft and Apple publish address feeds, and Amazon and Ahrefs now do too. Meta-ExternalAgent, Bytespider and CCBot publish none, so a request claiming those names cannot be checked.

### Why is an unverifiable claim worse than a forgery rate?

A forgery rate can be measured and acted on. An unfalsifiable claim cannot be confirmed or denied by anyone, so no amount of care on your side settles it.

### How do I verify an AI crawler is real?

Check the request's source address against what the operator publishes: an IP feed at a documented URL, or a reverse-DNS hostname you resolve and then forward-confirm. If the operator publishes neither, there is nothing to check against and the claim cannot be adjudicated either way.

### What does it mean when a crawler cannot be verified?

The operator publishes no address range, no reverse-DNS convention and no equivalent documentation, so a request with that name could be the operator or anyone else. It is not evidence of forgery; it is the absence of evidence in either direction.

### Should I block crawlers I cannot verify?

Blocking by name does not work on an identity you cannot verify, because the sender chooses the name. Use controls that do not depend on who is asking: per-address rate limits, path rules, size limits and challenges. Keep identity rules for operators that publish ranges.

### What does publishing a range cost an operator?

A JSON file at a stable URL and the discipline to keep it current. Leaving it out is a choice, not a technical limit.

## Related findings

[2026-10-07 Eight weeks of AI crawlers on one site: who was really who 24,627 verified crawler requests to crawlcheck.io in eight weeks: who was real, which names only came forged, and what verified ag](/blog/eight-weeks-of-agent-conduct-who-was-really-who)[2026-08-15 How much AI crawler traffic is forged? We check the IPs Requests claiming GPTBot, ClaudeBot or Googlebot, checked against each operator's published IP ranges: the live forged rate and ho](/blog/how-much-ai-crawler-traffic-is-forged)[2026-10-15 Agent conduct, October 2026: 34,676 verified requests Agent conduct on crawlcheck.io, 12 Aug to 15 Oct 2026: 34,676 verified requests, 9 of 12 crawlers read robots.txt, 3 never did, 84](/blog/agent-conduct-october-2026)

Where this is used

Guides that cite this finding:

- [AI crawler list: what each bot does and how to verify it](/guides/ai-crawler-list-what-each-one-does)
- [Do AI crawlers and agents respect robots.txt?](/guides/do-ai-crawlers-respect-robots-txt)
- [How to block AI crawlers like GPTBot and ClaudeBot](/guides/how-to-block-ai-crawlers)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [/](/) · [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) | **Directory:** [Deepcrawl](/directory/deepcrawl) | **Registry:** [botview.app](/registry/botview.app) · [dreamsoft4u.com](/registry/dreamsoft4u.com) · [jimbrownegmc.com](/registry/jimbrownegmc.com) · [paypal.com](/registry/paypal.com) · [woorank.com](/registry/woorank.com) · [agent-exec.448c.org](/registry/capabilities/agent-exec.448c.org) | **Datasets:** [West Virginia](/datasets/state/west-virginia)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /blog/the-crawlers-nobody-can-check.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.