Exclusive finding · Findings · 2026-08-15 · By VSNARY | Emmanuel Orta
How much AI crawler traffic is forged? We check the IPs
A user-agent string is a claim, not an identity. Every request here that names a known crawler is checked against the IP ranges its operator publishes, and the live share that fails is on this page.
A user-agent string such as GPTBot is a claim anyone can type, not an identity. Operators including OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity publish the IP ranges their crawlers use, so each claim can be sorted as verified (inside the range), spoofed (outside it) or unverifiable (no ranges published). On CrawlCheck's own first-party traffic a measurable share of checkable claims is spoofed; the rate on this page is recomputed from the live record. Verify by source address before you trust or block any crawler name.
A request arrives saying User-Agent: GPTBot. That is not an identity; it is a claim, typed by whoever sent the request. Most crawler statistics repeat the claim. This page adjudicates it: OpenAI, Anthropic, Google, Microsoft, Apple, Perplexity, Amazon and Ahrefs publish the IP ranges their crawlers send from, so a claim can be checked against the operator’s own list, and a measurable share fail.
How is a crawler claim checked?
Every request that names a known crawler lands in exactly one of three buckets, and keeping the third separate is the whole method:
- Verified: the source IP is inside the range the operator publishes.
- Spoofed: the operator publishes ranges and this IP is outside all of them. Someone else is wearing the name.
- Unverifiable: the operator publishes no range feed. This is not forgery. Meta-ExternalAgent, Bytespider, SemrushBot and DataForSeoBot sit here; their requests can be neither confirmed nor accused.
The forged rate divides spoofed claims by adjudicable claims only, where a published feed made a verdict possible. Folding unverifiable claims into either side produces a number that is wrong by construction, and a tool that quotes a forgery rate without saying how much of its traffic was unadjudicable has hidden its uncertainty inside a percentage. Which operators publish a feed and which do not is its own measurement, and it changes.
What is the forged rate right now?
The table and sentence below are recomputed from the live record each time this page loads, so a copy of the number ages and the URL does not.
Forged crawler identities #
11.7% of the requests that named themselves as a known crawler here were not that crawler. 24713 of 210891 checkable claims came from an address outside the range the operator publishes.
Forged rate by claimed identity — share of checkable claims from outside the operator’s published range
| Claimed to be | Claims | Verified | Forged | Unverifiable | Forged rate |
|---|---|---|---|---|---|
| Meta-ExternalAgent | 103620 | 0 | 0 | 103620 | no feed |
| Amazonbot | 87876 | 33918 | 8915 | 45044 | 20.8% |
| PerplexityBot | 35256 | 34927 | 334 | — | 0.9% |
| YandexBot | 30506 | 28070 | 20 | 2438 | 0.1% |
| SemrushBot | 29918 | 0 | 0 | 29920 | — |
| Googlebot | 22937 | 12236 | 10562 | 139 | 46.3% |
| Bytespider | 19702 | 0 | 0 | 19702 | no feed |
| AhrefsBot | 19197 | 14554 | 4 | 4640 | 0% |
| DataForSeoBot | 16743 | 0 | 0 | 16743 | no feed |
| ClaudeBot | 12975 | 12642 | 333 | — | 2.6% |
| Applebot | 12751 | 12452 | 299 | — | 2.3% |
| Bingbot | 11160 | 10853 | 168 | 139 | 1.5% |
A user-agent is a claim, not an identity. Verified means the source IP sits inside a range the operator publishes — OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish one. Unverifiable is not forgery: some operators publish no range at all, so their requests can be neither confirmed nor accused, and they are excluded from the rate rather than counted against it. This is traffic to this site only, and this site is small — it is a floor on the problem, not a survey of the web.
Citing this figure. The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged#the-rate. Sentence form, as of 2026-11-20: 11.7% of requests claiming a verifiable AI-crawler identity, measured by CrawlCheck on its own traffic (24713 of 210891 adjudicable claims), came from outside the operator's published IP ranges; the most-forged name was Google-Extended at 100%.
These are requests to CrawlCheck’s own endpoints and to sites it operates or watches: first-party observation, no third-party logs. It is not a random sample of the web, so quote it as “measured by CrawlCheck on its own traffic”, not as a web-wide figure. Earlier editions of this page led with a much higher September share. Part of that figure was our own tooling (below), the traffic mix has changed since, and the live rate above supersedes it.
Why does every per-crawler rate need a minimum?
One spoofed request out of one is a 100% forgery rate and evidence of nothing. So no crawler is named or charted until enough adjudicable claims sit behind it to be a pattern. That rule was not always applied evenly: the citation sentence refused to name a most-forged crawler on thin evidence, but the bar chart above it once had no minimum, and two names sat at the top at 100% on two claims each. The chart now uses the same rule as the sentence.
Why would anyone forge a crawler name?
Because the name opens doors. Sites allowlist Googlebot in rate limiters and paywalls, and increasingly allowlist the AI crawlers they want to be cited by. A scraper wearing GPTBot’s name inherits GPTBot’s welcome and hands OpenAI the blame. Some of it is plainly hostile: forged crawler names have been seen asking for .env files and SSH keys. Control tokens are the clearest case. Google never sends Google-Extended as a user agent, so every request carrying that name is someone else.
Our own extension was in the count
CrawlCheck’s browser extension sends requests under crawler names, because to see what a site serves GPTBot you have to ask as GPTBot. To our own telemetry that looked exactly like a stranger forging the name, so for weeks every scan by one of our users pushed up the forgery rate we published about everyone else.
We found it by the shape, not by reading code: a cluster of failed verifications on one path within seconds, wearing five different crawler names, a pattern no real crawler produces. The source traced back to our own network. The extension’s probes are now marked as tooling. The marker only changes how a failed check is labelled; it never turns a failed check into a pass, so it can only make our own published number smaller. Failures recorded before the change carry no such label, and they are kept in their own unassigned group rather than guessed into whichever column flatters us.
Group by address before you conclude anything
The rate is a property of claims, not machines. We nearly published a 57% crawler block rate from our own logs before grouping the refusals by source address: all of them came from one IP presenting seven crawler identities in a day, while the genuine crawlers in the same logs were refused zero times. Both errors were caught by grouping on something other than the claimed name. A user-agent is the one field in a request that carries no evidence about itself.
| Group claims by | What it exposes |
|---|---|
| Source IP | One machine wearing several crawler names; any address claiming more than one named crawler is forging at least one |
| Path requested | Credential and config probes (.env, .ssh) dressed as a crawler; tooling hitting report or API URLs rather than content |
| Time window | Bursts of different names within seconds, which no operator’s crawler produces |
| Verification bucket | How much of the log could not be checked at all, before you quote any rate |
If you build a check: let a marker label, never absolve
Any measurement that sees its own probes needs a way to recognise them, and the temptation is always to let that marker do one more useful thing. Every one of those things is a bypass, because a forger can send the same marker.
| A self-identifying marker may… | Allowed? | Why |
|---|---|---|
| Turn a failed verification into a pass | Never | Every forger would send it and vanish from the count |
| Change which category an already-failed check is filed under | Yes | It can only make your own published problem smaller |
| Exclude your own fetches from your traffic figures | Yes | Leaving them in would inflate every figure you publish |
| Suppress a finding about the site being measured | Never | Who asked has no bearing on what was served |
| Grant access, raise a grade or skip a limit | Never | Anything behind a header anyone can type is behind nothing |
The same applies to any signal the subject of a measurement controls: it may describe the measurement, never decide it. If you see a self-identifying header from someone else’s tool on an inbound request, treat it as decoration. Only the source address settles who sent it.
What should a site owner do?
- Never make an access decision on a user-agent alone. Robots rules key on the claimed name; firewall allowlists and log analysis should key on the verified identity.
- Check the address against the operator’s current feed. Ranges change, and a stale copy will call an operator’s new addresses spoofed. Fetch the feed fresh before judging a request.
- Do not rely on reverse DNS alone. Google’s hostname rule for Googlebot passes rented Google Cloud servers; check the address against the published crawler ranges as well.
- Report the unverifiable share beside any rate. Otherwise the rate is quietly measuring how many operators publish feeds.
Step-by-step instructions are in how to verify Googlebot, Bingbot and AI crawlers by IP, and every agent named here, with its operator and job, is in the AI crawler list.
What this measurement does not show
A verified request proves the address belongs to the operator, not that the operator behaved well. A spoofed request proves the address is outside the published range, not who sent it. That is why no single request here is ever published as an accusation against a named company: only rates, over a minimum, with the unadjudicable share stated beside them.
CrawlCheck’s dataset when this page loaded: 570,959 sites measured.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
How can you tell if AI crawler traffic is forged?
Check the request's source address against the IP ranges the named operator publishes. The user-agent is a claim anyone can type; only the published range makes that claim checkable. An address outside every published range is someone else wearing the name.
Why three outcomes instead of forged or genuine?
Because some operators publish no range at all. Requests in their name can be neither confirmed nor accused, and folding them into either bucket produces a rate that is wrong by construction. Keep them as unverifiable and state how many there were.
Is the forged rate a web-wide figure?
No. CrawlCheck measures it on first-party traffic to its own endpoints and the sites it operates or watches. It should be quoted as measured by CrawlCheck on its own traffic, with the date, not as a property of the whole web.
Why would someone forge GPTBot or Googlebot?
Because sites allowlist those names in rate limiters, paywalls and firewalls. A scraper wearing a welcomed crawler's name inherits the welcome, and the real operator takes the blame. Some forged requests go straight for credential files such as .env and SSH keys.
What should a site owner do with a forgery rate?
Treat it as a reason to verify by address before trusting or blocking by name. Blocking a user-agent string stops the honest traffic and none of the forged traffic, and allowlisting one lets every forger through.
Can a self-identifying header prove a request is harmless?
No. Anyone can send any header. A marker can label a request for the sender's own bookkeeping, but it must never be able to clear a verification check, because then the check measures nothing a forger cannot fake.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.