Guides · 2026-11-20 · By VSNARY | Emmanuel Orta
How to tell a real AI crawler from one borrowing its name
A user agent is a claim anyone can type. Three checks settle it: reverse DNS with a forward confirm, the operator's published address list, and signed requests. Here is how to run each on your own logs.
A request that names an AI crawler is only making a claim. To tell a real crawler from one borrowing its name, check where the request came from, not what it calls itself: confirm the address with a reverse DNS lookup and a matching forward lookup, find it in the address list the operator publishes, or verify a signed request against the operator's published key. On 20 November 2026, 11.6% of requests that named a checkable crawler on CrawlCheck's own traffic (24,717 of 213,068) came from outside the operator's published range.
Why the name in a request proves nothing #
Every request to your site carries a user agent, the name a client gives itself in a request header. A crawler that collects pages for an AI model puts its product name there, and most server logs, analytics tools and firewall rules read that name and stop. The trouble is that the header is typed by the sender. Any script on any rented server can send the exact name of any crawler in the world, and nothing in the protocol stops it.
People borrow crawler names because the names open doors. Sites allowlist the crawlers they want to be cited or indexed by, exempt them from rate limits, and sometimes let them past paywalls and bot challenges. A scraper wearing a trusted name inherits that welcome, and the real operator takes the blame for whatever the scraper does. Some of the borrowed traffic is plainly hostile: forged crawler names have been seen asking for configuration files and private keys.
So the question to ask of a request is not "what does it say it is" but "can the place it came from prove it". This guide covers the three ways to answer that, what each one proves, where each one fails, and how to run them over your own access logs.
How often the name is borrowed #
On 20 November 2026, 11.6% of requests that named a checkable crawler on CrawlCheck's own traffic (24,717 of 213,068 claims) came from an address outside the range that crawler's operator publishes. A checkable claim is one whose operator publishes an address list, so a verdict is possible at all. The figure is recomputed every time the page loads, so follow the live forged-claims rate rather than copying the number; it will have moved by the time you read this.
Two cautions on reading that figure. It is first-party traffic to CrawlCheck and the sites it operates, not a survey of the web, so treat it as evidence that the problem exists rather than a rate your site will see. And the share varies enormously by name: on the same page some crawler names are almost never forged while others are forged on a large share of their claims. The only rate that matters for your decisions is the one in your own logs.
The second number worth knowing is how many names can be checked at all. On 20 November 2026 the AI crawlers reference listed 116 named agents resolved from robots.txt files, of which 47 were verifiable against a published address range. For the rest, no amount of care on your side can turn a claim into a verdict, and the honest label is unverifiable.
Three outcomes, never two #
Every request that names a crawler should land in exactly one of three buckets:
- Verified: the source address passes a check the operator controls (its DNS, its published list or its signing key).
- Forged: the operator offers a way to check, and this address fails it. Someone else is using the name.
- Unverifiable: the operator offers no way to check. The request might be real or might not, and you cannot tell.
The mistake most log reports make is collapsing the third bucket into one of the first two. Counting unverifiable requests as forged accuses operators of nothing worse than not publishing a list. Counting them as verified lets anyone in who picks a name from the unverifiable group. When you quote a forged share, divide forged claims by verified plus forged claims only, and say separately how many claims could not be judged.
The three checks compared #
| Check | What you need | What a pass proves | Where it fails |
|---|---|---|---|
| Reverse DNS with forward confirm | The source address and the hostname suffix the operator documents | The address belongs to a host in the operator's own domain | Operators that document no hostname; slow lookups at request time; resolvers that time out |
| Published address list | The operator's list of address ranges, refreshed regularly | The address sits inside a range the operator says it sends from | Operators that publish no list; stale copies of the list; shared cloud ranges if the list is broad |
| Signed request | The signature headers and the operator's published public key | The request was signed by a key the operator controls, for your host, recently | Most crawlers do not sign yet; a missing signature proves nothing |
| User agent only | The header | Nothing about identity | Always forgeable |
The checks stack. Use the strongest one the operator supports, and fall back to the next. If an operator offers none, the request stays unverifiable however polite its name.
Check 1: reverse DNS with a forward confirm #
Reverse DNS asks the address's owner what hostname it belongs to. On its own that answer is worthless, because whoever controls an address can set its reverse record to any name they like, including a crawler operator's. The forward confirm closes that hole. The steps:
- Take the source address from your log (the connecting address, not a forwarded header).
- Run a reverse lookup on it and read the hostname that comes back.
- Check that the hostname ends in the domain the operator documents for its crawler. Match the whole suffix after a dot, not a substring, or a hostname like
operator.example.attacker.netpasses. - Run a forward lookup on that hostname and check that the original address is among the answers.
Only a pass on all four steps is a verified request. The forward step works because only the operator can publish address records inside its own domain. Cache results by address for a day or so; a busy site can otherwise make thousands of lookups an hour, and a resolver timeout should leave the request unverified, never forged.
Check 2: the operator's published address list #
Many AI crawler operators do not document a hostname at all. Instead they publish a machine-readable list of address ranges, usually a JSON file of CIDR blocks. CIDR is the short notation for a range of addresses, a starting address plus how many leading bits are fixed. The check is simple: fetch the list, and test whether the source address falls inside any range in it.
Three details decide whether this check is right in practice. First, refresh the list on a schedule, daily is reasonable, because operators add and retire ranges and a stale copy will call new real traffic forged. Second, handle both IPv4 and IPv6 ranges; plenty of crawler traffic arrives over IPv6 and a parser that skips those lines turns real requests into forgeries. Third, keep one list per crawler name rather than one merged list per operator, because an operator's different crawlers can send from different ranges, and an address valid for one name may not be valid for another.
The AI crawlers reference records, for each named agent, whether its operator publishes a list and where. The per-operator verification guide walks through the lookup for several of the major operators step by step.
Check 3: signed requests #
The newest method does not depend on addresses at all. A crawler signs each request with a private key, using HTTP Message Signatures, the standard published as RFC 9421, and publishes the matching public key in a key directory on its own domain, at the well-known path /.well-known/http-message-signatures-directory. A request then carries a Signature header, a Signature-Input header that lists which parts of the request were signed and when, and often a Signature-Agent header naming the domain whose directory holds the key.
To verify one, fetch the key directory from the operator's domain over HTTPS, find the key named in the signature input, and check the signature over the listed components. Then check the parts that stop replay: the signature should cover your host name, the creation time should be recent, and the expiry should not have passed. A valid signature proves the request was made by someone holding the operator's key, which is stronger than any address check because it survives the operator changing networks.
The limit is adoption. Signing is optional, and most crawlers do not do it yet, so a request without a signature is not evidence of forgery. Treat a valid signature as verified, an invalid one as forged, and a missing one as a reason to fall back to the address checks.
What to check in your own logs #
You do not need special tools to start. Any access log that records the connecting address, the time, the path and the user agent is enough. A workable first pass:
- List the claims. Pull every line whose user agent contains a crawler name you care about, and group by name.
- Group each name by source address. A real crawler usually arrives from a modest number of ranges. One name spread across hundreds of unrelated networks is a pattern worth a closer look.
- Judge each address once. Run the strongest check the operator supports, cache the verdict, and label every line from that address with it.
- Read what the forged lines asked for. Paths like configuration files, admin panels or backup archives under a crawler's name are not crawling. Paths your robots.txt disallows are worth noting on verified lines too, because that is the real crawler's conduct, not an impostor's.
- Report three counts, not one. Verified, forged and unverifiable, per name, with the forged share computed over the first two only.
A one-line starting point on a standard combined log format, which counts crawler-looking requests per source address so you know which addresses to check first: grep -iE "bot|crawler|agent" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -50. Mask or truncate addresses before you paste results anywhere public; a full address in a screenshot is personal data in many places.
Mistakes that turn a good check into a wrong verdict #
- Trusting a forwarded header. If your site sits behind a proxy or CDN, the connecting address is the proxy's. Use the address your edge records for the original client, and never one taken from a header the client itself can set.
- Substring matching on names. A user agent that contains a crawler's name inside a longer string is not necessarily claiming to be that crawler. Match the product token the operator documents.
- Counting your own tools. SEO audits, uptime monitors and testing tools often send requests under crawler names so they can see what a crawler is served. CrawlCheck's own forged figure once counted requests from its own browser extension as strangers forging names; the forged-traffic page tells that story and now labels those requests as tooling. Know what your own tools send before you count anyone else.
- Stale lists and cached verdicts kept forever. Ranges move. A verdict from last month is a guess today.
- Blocking on the name alone. A rule that blocks a crawler name blocks the real crawler and every impostor equally, and a rule that allows it lets every impostor in.
What to do with each verdict #
Verified requests are the real crawler, so your robots.txt policy is the right control: allow it, disallow paths, or block it entirely, and it should comply. Forged requests can be blocked or rate limited at your edge with no cost to the real crawler, because the real one never sends from those addresses. Unverifiable requests should be treated as what they provably are, an ordinary visitor with a name you cannot confirm: apply the same limits and challenges you apply to any other unknown client.
A verified result answers one question only: the request came from the operator's network or key. It does not tell you what the operator will do with the page, whether it respects your robots.txt, or how fast it will crawl. Those are separate measurements, and the AI crawlers reference records them per agent where they have been measured.
CrawlCheck’s dataset when this page loaded: 582,334 sites measured.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
Can a request fake an AI crawler's user agent?
Yes. The user agent is a header the sender types, so any script on any server can send the exact name of any crawler. The name alone proves nothing about where the request came from. Only a check the operator controls, such as its DNS, its published address list or its signing key, can confirm the identity.
What is reverse DNS with a forward confirm?
It is a two-step address check. A reverse lookup turns the source address into a hostname, and you confirm the hostname ends in the operator's documented domain. A forward lookup on that hostname must then return the original address. The forward step matters because only the operator can publish address records inside its own domain.
What if a crawler's operator publishes no address list or hostname?
Then its requests are unverifiable: they can be neither confirmed nor accused. Do not count them as forged, and do not treat them as verified. Handle them like any other unknown visitor, with the same limits and challenges you apply to ordinary traffic.
How common is a borrowed crawler name?
On 20 November 2026, 11.6% of requests that named a checkable crawler on CrawlCheck's own traffic (24,717 of 213,068 claims) came from outside the operator's published address range. That figure is first-party traffic, not a survey of the web, and the live rate is recomputed on CrawlCheck's forged-traffic page.
What is a signed crawler request?
A request signed with HTTP Message Signatures (RFC 9421). The crawler signs parts of each request with a private key and publishes the public key in a key directory on its own domain. A valid signature that covers your host and is recent proves the request was made by the key holder. A missing signature proves nothing, because signing is still optional.
Should I block a crawler by its user agent?
Not on the name alone. A name-based block stops the real crawler and every impostor equally, and a name-based allow lets every impostor in. Verify first: use robots.txt to control verified crawlers, and block or rate limit forged requests at your edge, which costs the real crawler nothing.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.