CrawlCheck

Agent view · Findings · 2026-08-15

How much AI crawler traffic is forged? We check the IPs

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes22,033
words3,281
sha-25656c1787fe6aa37446af177b44d2f01bede26b6f94d3ec51c0b9cf23e9312a30d
repr-digestsha-256=:VsF4f+aqN0Rq8Xe0TS8Bvt4mtvlNPsUcC5zyPpMSow0=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795182383;expires=1795185983;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="yQ7mJlBviAcMPWWnodeWvg";tag="crawlcheck-page"
signaturesig1=:vxAL24sM5EUBDU6cTBcoTFvkbn6sWyPt5bju9Y8Y1tur3QXLNI+VOONU5gGw/2yGfEiwAI/Mo9wBDAB314uPAA==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Exclusive finding · Findings · 2026-08-15 · By [VSNARY | Emmanuel Orta](/author)

# How much AI crawler traffic is forged? We check the IPs

A user-agent string is a claim, not an identity. Every request here that names a known crawler is checked against the IP ranges its operator publishes, and the live share that fails is on this page.

A user-agent string such as GPTBot is a claim anyone can type, not an identity. Operators including OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity publish the IP ranges their crawlers use, so each claim can be sorted as verified (inside the range), spoofed (outside it) or unverifiable (no ranges published). On CrawlCheck's own first-party traffic a measurable share of checkable claims is spoofed; the rate on this page is recomputed from the live record. Verify by source address before you trust or block any crawler name.

CrawlCheck exclusive finding

Sample 3,415 checkable crawler claims across sites we operate, sorted verified / spoofed / unverifiable

Measured September 2026, read from a live record

What it established The three-outcome method for crawler claims (verified, spoofed, unverifiable), with the forged share recomputed from the live record on every read

One of [23 findings measured only here](/blog?kind=exclusive). Every figure comes from CrawlCheck’s own readings, dated, with the method at the foot. Have your own? [Submit a finding](/submit-finding).

Later readings

The rate here is from sites we operate. On crawlcheck.io itself the forged share was 3.1% over eight weeks (787 of 25,414 named claims, [by crawler](/blog/eight-weeks-of-agent-conduct-who-was-really-who)) and 2.4% in the [October conduct edition](/blog/agent-conduct-october-2026) (842 requests). Same method, different sites: how much of a site’s crawler traffic is forged depends on the site, which is why the check has to run on your own logs.

On this page

- How is a crawler claim checked?
- What is the forged rate right now?
- Forged crawler identities
- Why does every per-crawler rate need a minimum?
- Why would anyone forge a crawler name?
- Our own extension was in the count
- Group by address before you conclude anything
- If you build a check: let a marker label, never absolve
- What should a site owner do?
- What this measurement does not show

A request arrives saying `User-Agent: GPTBot`. That is not an identity; it is a claim, typed by whoever sent the request. Most crawler statistics repeat the claim. This page adjudicates it: OpenAI, Anthropic, Google, Microsoft, Apple, Perplexity, Amazon and Ahrefs publish the IP ranges their crawlers send from, so a claim can be checked against the operator’s own list, and a measurable share fail.

## How is a crawler claim checked?

Every request that names a known crawler lands in exactly one of three buckets, and keeping the third separate is the whole method:

- **Verified**: the source IP is inside the range the operator publishes.
- **Spoofed**: the operator publishes ranges and this IP is outside all of them. Someone else is wearing the name.
- **[Unverifiable](/glossary/unverifiable)**: the operator publishes no range feed. This is *not* forgery. Meta-ExternalAgent, [Bytespider](/glossary/bytespider), SemrushBot and DataForSeoBot sit here; their requests can be neither confirmed nor accused.

Every claim lands in exactly one of three buckets. The forged rate is spoofed divided by verified plus spoofed; unverifiable claims are never in it.
The forged rate divides spoofed claims by *adjudicable* claims only, where a published feed made a verdict possible. Folding unverifiable claims into either side produces a number that is wrong by construction, and a tool that quotes a forgery rate without saying how much of its traffic was unadjudicable has hidden its uncertainty inside a percentage. [Which operators publish a feed and which do not](/blog/the-crawlers-nobody-can-check) is its own measurement, and it changes.

## What is the forged rate right now?

The table and sentence below are recomputed from the live record each time this page loads, so a copy of the number ages and the [URL](/glossary/url) does not.

## Forged crawler identities #

**11.7% of the requests that named themselves as a known crawler here were not that crawler.** 24713 of 211271 checkable claims came from an address outside the range the operator publishes.

Forged rate by claimed identity — share of checkable claims from outside the operator’s published range

| Claimed to be | Claims | Verified | Forged | Unverifiable | Forged rate |
| --- | --- | --- | --- | --- | --- |
| Meta-ExternalAgent | 103620 | 0 | 0 | 103620 | no feed |
| Amazonbot | 87931 | 33973 | 8915 | 45044 | 20.8% |
| PerplexityBot | 35256 | 34927 | 334 | — | 0.9% |
| YandexBot | 30509 | 28073 | 20 | 2438 | 0.1% |
| SemrushBot | 29921 | 0 | 0 | 29923 | — |
| Googlebot | 22938 | 12237 | 10562 | 139 | 46.3% |
| Bytespider | 19702 | 0 | 0 | 19702 | no feed |
| AhrefsBot | 19197 | 14554 | 4 | 4640 | 0% |
| DataForSeoBot | 16743 | 0 | 0 | 16743 | no feed |
| ClaudeBot | 13290 | 12957 | 333 | — | 2.5% |
| Applebot | 12752 | 12453 | 299 | — | 2.3% |
| Bingbot | 11162 | 10855 | 168 | 139 | 1.5% |

A user-agent is a claim, not an identity. Verified means the source IP sits inside a range the operator publishes — OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish one. **Unverifiable is not forgery:** some operators publish no range at all, so their requests can be neither confirmed nor accused, and they are excluded from the rate rather than counted against it. This is traffic to this site only, and this site is small — it is a floor on the problem, not a survey of the web.

**Citing this figure.** The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: `https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged#the-rate`. Sentence form, as of 2026-11-20: *11.7% of requests claiming a verifiable AI-crawler identity, measured by CrawlCheck on its own traffic (24713 of 211271 adjudicable claims), came from outside the operator's published IP ranges; the most-forged name was [Google-Extended](/glossary/google-extended) at 100%.*

These are requests to CrawlCheck’s own endpoints and to sites it operates or watches: first-party observation, no third-party logs. It is **not** a random sample of the web, so quote it as “measured by CrawlCheck on its own traffic”, not as a web-wide figure. Earlier editions of this page led with a much higher September share. Part of that figure was our own tooling (below), the traffic mix has changed since, and the live rate above supersedes it.

## Why does every per-crawler rate need a minimum?

One spoofed request out of one is a 100% forgery rate and evidence of nothing. So no crawler is named or charted until enough adjudicable claims sit behind it to be a pattern. That rule was not always applied evenly: the [citation](/glossary/citation) sentence refused to name a most-forged crawler on thin evidence, but the bar chart above it once had no minimum, and two names sat at the top at 100% on two claims each. The chart now uses the same rule as the sentence.

## Why would anyone forge a crawler name?

Because the name opens doors. Sites allowlist Googlebot in rate limiters and paywalls, and increasingly allowlist the AI crawlers they want to be cited by. A scraper wearing GPTBot’s name inherits GPTBot’s welcome and hands OpenAI the blame. Some of it is plainly hostile: [forged crawler names have been seen asking for .env files and SSH keys](/blog/the-ai-crawler-that-wanted-your-env-file). Control tokens are the clearest case. Google never sends Google-Extended as a [user agent](/glossary/user-agent), so [every request carrying that name](/blog/a-name-google-says-it-never-sends) is someone else.

## Our own extension was in the count

CrawlCheck’s browser extension sends requests under crawler names, because to see what a site serves GPTBot you have to ask as GPTBot. To our own telemetry that looked exactly like a stranger forging the name, so for weeks every scan by one of our users pushed up the forgery rate we published about everyone else.

We found it by the shape, not by reading code: a cluster of failed verifications on one path within seconds, wearing five different crawler names, a pattern no real crawler produces. The source traced back to our own network. The extension’s probes are now marked as tooling. The marker only changes how a failed check is labelled; it never turns a failed check into a pass, so it can only make our own published number smaller. Failures recorded before the change carry no such label, and they are kept in their own unassigned group rather than guessed into whichever column flatters us.

## Group by address before you conclude anything

The rate is a property of claims, not machines. [We nearly published a 57% crawler block rate](/blog/we-almost-published-that-our-host-was-blocking-claude) from our own logs before grouping the refusals by source address: all of them came from one IP presenting seven crawler identities in a day, while the genuine crawlers in the same logs were refused zero times. Both errors were caught by grouping on something other than the claimed name. A user-agent is the one field in a request that carries no evidence about itself.

| Group claims by | What it exposes |
| --- | --- |
| Source IP | One machine wearing several crawler names; any address claiming more than one named crawler is forging at least one |
| Path requested | Credential and config probes (.env, .ssh) dressed as a crawler; tooling hitting report or API URLs rather than content |
| Time window | Bursts of different names within seconds, which no operator’s crawler produces |
| Verification bucket | How much of the log could not be checked at all, before you quote any rate |

## If you build a check: let a marker label, never absolve

Any measurement that sees its own probes needs a way to recognise them, and the temptation is always to let that marker do one more useful thing. Every one of those things is a bypass, because a forger can send the same marker.

| A self-identifying marker may… | Allowed? | Why |
| --- | --- | --- |
| Turn a failed verification into a pass | **Never** | Every forger would send it and vanish from the count |
| Change which category an already-failed check is filed under | Yes | It can only make your own published problem smaller |
| Exclude your own fetches from your traffic figures | Yes | Leaving them in would inflate every figure you publish |
| Suppress a finding about the site being measured | **Never** | Who asked has no bearing on what was served |
| Grant access, raise a grade or skip a limit | **Never** | Anything behind a header anyone can type is behind nothing |

The same applies to any signal the subject of a measurement controls: it may describe the measurement, never decide it. If you see a self-identifying header from someone else’s tool on an inbound request, treat it as decoration. Only the source address settles who sent it.

## What should a site owner do?

- **Never make an access decision on a user-agent alone.** Robots rules key on the claimed name; firewall allowlists and log analysis should key on the verified identity.
- **Check the address against the operator’s current feed.** Ranges change, and a stale copy will call an operator’s new addresses spoofed. Fetch the feed fresh before judging a request.
- **Do not rely on reverse DNS alone.** [Google’s hostname rule for Googlebot passes rented Google Cloud servers](/blog/googles-googlebot-verification-rule-passes-any-google-cloud-server); check the address against the [published crawler ranges](/glossary/published-crawler-ranges) as well.
- **Report the unverifiable share beside any rate.** Otherwise the rate is quietly measuring how many operators publish feeds.

Step-by-step instructions are in [how to verify Googlebot, Bingbot and AI crawlers by IP](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers), and every agent named here, with its operator and job, is in [the AI crawler list](/guides/ai-crawler-list-what-each-one-does).

## What this measurement does not show

A verified request proves the address belongs to the operator, not that the operator behaved well. A spoofed request proves the address is outside the published range, not who sent it. That is why no single request here is ever published as an accusation against a named company: only rates, over a minimum, with the unadjudicable share stated beside them.

CrawlCheck’s dataset when this page loaded: 570,959 sites measured.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fhow-much-ai-crawler-traffic-is-forged%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fhow-much-ai-crawler-traffic-is-forged%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=How%20much%20AI%20crawler%20traffic%20is%20forged%3F%20We%20check%20the%20IPs)[Email](mailto:?subject=How%20much%20AI%20crawler%20traffic%20is%20forged%3F%20We%20check%20the%20IPs&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fhow-much-ai-crawler-traffic-is-forged)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### How can you tell if AI crawler traffic is forged?

Check the request's source address against the IP ranges the named operator publishes. The user-agent is a claim anyone can type; only the published range makes that claim checkable. An address outside every published range is someone else wearing the name.

### Why three outcomes instead of forged or genuine?

Because some operators publish no range at all. Requests in their name can be neither confirmed nor accused, and folding them into either bucket produces a rate that is wrong by construction. Keep them as unverifiable and state how many there were.

### Is the forged rate a web-wide figure?

No. CrawlCheck measures it on first-party traffic to its own endpoints and the sites it operates or watches. It should be quoted as measured by CrawlCheck on its own traffic, with the date, not as a property of the whole web.

### Why would someone forge GPTBot or Googlebot?

Because sites allowlist those names in rate limiters, paywalls and firewalls. A scraper wearing a welcomed crawler's name inherits the welcome, and the real operator takes the blame. Some forged requests go straight for credential files such as .env and SSH keys.

### What should a site owner do with a forgery rate?

Treat it as a reason to verify by address before trusting or blocking by name. Blocking a user-agent string stops the honest traffic and none of the forged traffic, and allowlisting one lets every forger through.

### Can a self-identifying header prove a request is harmless?

No. Anyone can send any header. A marker can label a request for the sender's own bookkeeping, but it must never be able to clear a verification check, because then the check measures nothing a forger cannot fake.

## Related findings

[2026-10-02 How to verify Googlebot, GPTBot and other crawlers by IP Verify Googlebot, Bingbot, GPTBot and ClaudeBot by reverse DNS or the operator's IP list. The file for each crawler, the traps, an](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers)[2026-10-07 Eight weeks of AI crawlers on one site: who was really who 24,627 verified crawler requests to crawlcheck.io in eight weeks: who was real, which names only came forged, and what verified ag](/blog/eight-weeks-of-agent-conduct-who-was-really-who)[2026-10-08 AI agents acting for people: 613 visits, one read robots.txt ChatGPT-User made 526 verified visits, Claude-User 79, Perplexity-User 8. Only Claude-User read robots.txt; none fetched a disallo](/blog/agents-acting-for-people-613-verified-visits-one-read-robots-txt)

Where this is used

Guides that cite this finding:

- [AI crawler list: what each bot does and how to verify it](/guides/ai-crawler-list-what-each-one-does)
- [Do AI crawlers and agents respect robots.txt?](/guides/do-ai-crawlers-respect-robots-txt)
- [How to block AI crawlers like GPTBot and ClaudeBot](/guides/how-to-block-ai-crawlers)
- [How to measure AI visibility yourself: three checks](/guides/how-to-measure-ai-visibility)
- [How to rank in ChatGPT: the part that is in your control](/guides/how-to-rank-in-chatgpt)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [docs.nginx.com](/registry/docs.nginx.com) · [insignialandscape.com](/registry/insignialandscape.com) · [orgintel360.com](/registry/orgintel360.com) · [vk.com](/registry/vk.com) · [Trade: Tucson](/registry/trade/roofing/az/tucson) | **Datasets:** [New Mexico](/datasets/state/new-mexico) · [Technologies: Tailwind CSS](/datasets/technologies/tailwind-css) · [Problems: LLMs link broken](/datasets/problems/llms-link-broken)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /blog/how-much-ai-crawler-traffic-is-forged.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.