CrawlCheck

Agent view · Guides · 2026-11-20

How to tell a real AI crawler from one borrowing its name

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/how-to-tell-a-real-ai-crawler-from-one-borrowing-its-name: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/how-to-tell-a-real-ai-crawler-from-one-borrowing-its-name>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes22,717
words3,471
sha-25668ed89e3ddfca007da7ef7fe5fdb68b64d7e95d9993c142371c1f466a7acf7a5
repr-digestsha-256=:aO2J4938oAfafvf+X9totk1+ldmZPBQjccH0Zqes96U=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311496;expires=1795315096;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="uFSjUJmV2slT1oqexxQ33g";tag="crawlcheck-page"
signaturesig1=:BkvhuipLKEkirYu+zmOwktXuFpzcdAkdoercdhCYATaCleU9m+6AFjA4NJHii22zL2+McS7o/e2DeApcFMbuBQ==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-11-20 · By [VSNARY | Emmanuel Orta](/author)

# How to tell a real AI crawler from one borrowing its name

A user agent is a claim anyone can type. Three checks settle it: reverse DNS with a forward confirm, the operator's published address list, and signed requests. Here is how to run each on your own logs.

A request that names an AI crawler is only making a claim. To tell a real crawler from one borrowing its name, check where the request came from, not what it calls itself: confirm the address with a reverse DNS lookup and a matching forward lookup, find it in the address list the operator publishes, or verify a signed request against the operator's published key. On 20 November 2026, 11.6% of requests that named a checkable crawler on CrawlCheck's own traffic (24,717 of 213,068) came from outside the operator's published range.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- Why the name in a request proves nothing
- How often the name is borrowed
- Three outcomes, never two
- The three checks compared
- Check 1: reverse DNS with a forward confirm
- Check 2: the operator's published address list
- Check 3: signed requests
- What to check in your own logs
- Mistakes that turn a good check into a wrong verdict
- What to do with each verdict

## Why the name in a request proves nothing #

Every request to your site carries a **[user agent](/glossary/user-agent)**, the name a client gives itself in a request header. A crawler that collects pages for an AI model puts its product name there, and most server logs, analytics tools and firewall rules read that name and stop. The trouble is that the header is typed by the sender. Any script on any rented server can send the exact name of any crawler in the world, and nothing in the protocol stops it.

People borrow crawler names because the names open doors. Sites allowlist the crawlers they want to be cited or indexed by, exempt them from rate limits, and sometimes let them past paywalls and bot challenges. A scraper wearing a trusted name inherits that welcome, and the real operator takes the blame for whatever the scraper does. Some of the borrowed traffic is plainly hostile: forged crawler names have been seen asking for configuration files and private keys.

So the question to ask of a request is not "what does it say it is" but "can the place it came from prove it". This guide covers the three ways to answer that, what each one proves, where each one fails, and how to run them over your own access logs.

## How often the name is borrowed #

**On 20 November 2026, 11.6% of requests that named a checkable crawler on CrawlCheck's own traffic (24,717 of 213,068 claims) came from an address outside the range that crawler's operator publishes.** A *checkable* claim is one whose operator publishes an address list, so a verdict is possible at all. The figure is recomputed every time the page loads, so follow [the live forged-claims rate](/blog/how-much-ai-crawler-traffic-is-forged#the-rate) rather than copying the number; it will have moved by the time you read this.

Two cautions on reading that figure. It is first-party traffic to CrawlCheck and the sites it operates, not a survey of the web, so treat it as evidence that the problem exists rather than a rate your site will see. And the share varies enormously by name: on the same page some crawler names are almost never forged while others are forged on a large share of their claims. The only rate that matters for your decisions is the one in your own logs.

The second number worth knowing is how many names can be checked at all. On 20 November 2026 the [AI crawlers reference](/ai-crawlers) listed 116 named agents resolved from [robots.txt](/glossary/robots-txt) files, of which 47 were verifiable against a published address range. For the rest, no amount of care on your side can turn a claim into a verdict, and the honest label is [unverifiable](/glossary/unverifiable).

## Three outcomes, never two #

Every request that names a crawler should land in exactly one of three buckets:

- **Verified**: the source address passes a check the operator controls (its DNS, its published list or its signing key).
- **Forged**: the operator offers a way to check, and this address fails it. Someone else is using the name.
- **Unverifiable**: the operator offers no way to check. The request might be real or might not, and you cannot tell.

The mistake most log reports make is collapsing the third bucket into one of the first two. Counting unverifiable requests as forged accuses operators of nothing worse than not publishing a list. Counting them as verified lets anyone in who picks a name from the unverifiable group. When you quote a forged share, divide forged claims by verified plus forged claims only, and say separately how many claims could not be judged.

## The three checks compared #

| Check | What you need | What a pass proves | Where it fails |
| --- | --- | --- | --- |
| Reverse DNS with forward confirm | The source address and the hostname suffix the operator documents | The address belongs to a host in the operator's own domain | Operators that document no hostname; slow lookups at request time; resolvers that time out |
| Published address list | The operator's list of address ranges, refreshed regularly | The address sits inside a range the operator says it sends from | Operators that publish no list; stale copies of the list; shared cloud ranges if the list is broad |
| Signed request | The signature headers and the operator's published public key | The request was signed by a key the operator controls, for your host, recently | Most crawlers do not sign yet; a missing signature proves nothing |
| User agent only | The header | Nothing about identity | Always forgeable |

The checks stack. Use the strongest one the operator supports, and fall back to the next. If an operator offers none, the request stays unverifiable however polite its name.

## Check 1: reverse DNS with a forward confirm #

Reverse DNS asks the address's owner what hostname it belongs to. On its own that answer is worthless, because whoever controls an address can set its reverse record to any name they like, including a crawler operator's. The forward confirm closes that hole. The steps:

- Take the source address from your log (the connecting address, not a forwarded header).
- Run a reverse lookup on it and read the hostname that comes back.
- Check that the hostname ends in the domain the operator documents for its crawler. Match the whole suffix after a dot, not a substring, or a hostname like `operator.example.attacker.net` passes.
- Run a forward lookup on that hostname and check that the original address is among the answers.

Only a pass on all four steps is a verified request. The forward step works because only the operator can publish address records inside its own domain. Cache results by address for a day or so; a busy site can otherwise make thousands of lookups an hour, and a resolver timeout should leave the request unverified, never forged.

## Check 2: the operator's published address list #

Many AI crawler operators do not document a hostname at all. Instead they publish a machine-readable list of address ranges, usually a JSON file of CIDR blocks. CIDR is the short notation for a range of addresses, a starting address plus how many leading bits are fixed. The check is simple: fetch the list, and test whether the source address falls inside any range in it.

Three details decide whether this check is right in practice. First, refresh the list on a schedule, daily is reasonable, because operators add and retire ranges and a stale copy will call new real traffic forged. Second, handle both IPv4 and IPv6 ranges; plenty of crawler traffic arrives over IPv6 and a parser that skips those lines turns real requests into forgeries. Third, keep one list per crawler name rather than one merged list per operator, because an operator's different crawlers can send from different ranges, and an address valid for one name may not be valid for another.

The [AI crawlers reference](/ai-crawlers) records, for each named agent, whether its operator publishes a list and where. The [per-operator verification guide](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers) walks through the lookup for several of the major operators step by step.

## Check 3: signed requests #

The newest method does not depend on addresses at all. A crawler signs each request with a private key, using [HTTP Message Signatures](/glossary/http-message-signatures), the standard published as RFC 9421, and publishes the matching public key in a key directory on its own domain, at the well-known path `/.well-known/http-message-signatures-directory`. A request then carries a `Signature` header, a `Signature-Input` header that lists which parts of the request were signed and when, and often a `Signature-Agent` header naming the domain whose directory holds the key.

To verify one, fetch the key directory from the operator's domain over HTTPS, find the key named in the signature input, and check the signature over the listed components. Then check the parts that stop replay: the signature should cover your host name, the creation time should be recent, and the expiry should not have passed. A valid signature proves the request was made by someone holding the operator's key, which is stronger than any address check because it survives the operator changing networks.

The limit is adoption. Signing is optional, and most crawlers do not do it yet, so a request without a signature is not evidence of forgery. Treat a valid signature as verified, an invalid one as forged, and a missing one as a reason to fall back to the address checks.

## What to check in your own logs #

You do not need special tools to start. Any access log that records the connecting address, the time, the path and the user agent is enough. A workable first pass:

- **List the claims.** Pull every line whose user agent contains a crawler name you care about, and group by name.
- **Group each name by source address.** A real crawler usually arrives from a modest number of ranges. One name spread across hundreds of unrelated networks is a pattern worth a closer look.
- **Judge each address once.** Run the strongest check the operator supports, cache the verdict, and label every line from that address with it.
- **Read what the forged lines asked for.** Paths like configuration files, admin panels or backup archives under a crawler's name are not crawling. Paths your robots.txt disallows are worth noting on verified lines too, because that is the real crawler's conduct, not an impostor's.
- **Report three counts, not one.** Verified, forged and unverifiable, per name, with the forged share computed over the first two only.

A one-line starting point on a standard combined log format, which counts crawler-looking requests per source address so you know which addresses to check first: `grep -iE "bot|crawler|agent" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -50`. Mask or truncate addresses before you paste results anywhere public; a full address in a screenshot is personal data in many places.

## Mistakes that turn a good check into a wrong verdict #

- **Trusting a forwarded header.** If your site sits behind a proxy or [CDN](/glossary/cdn), the connecting address is the proxy's. Use the address your edge records for the original client, and never one taken from a header the client itself can set.
- **Substring matching on names.** A user agent that contains a crawler's name inside a longer string is not necessarily claiming to be that crawler. Match the product [token](/glossary/token) the operator documents.
- **Counting your own tools.** SEO audits, uptime monitors and testing tools often send requests under crawler names so they can see what a crawler is served. CrawlCheck's own forged figure once counted requests from its own browser extension as strangers forging names; the forged-traffic page tells that story and now labels those requests as tooling. Know what your own tools send before you count anyone else.
- **Stale lists and cached verdicts kept forever.** Ranges move. A verdict from last month is a guess today.
- **Blocking on the name alone.** A rule that blocks a crawler name blocks the real crawler and every impostor equally, and a rule that allows it lets every impostor in.

## What to do with each verdict #

**Verified** requests are the real crawler, so your robots.txt policy is the right control: allow it, disallow paths, or block it entirely, and it should comply. **Forged** requests can be blocked or rate limited at your edge with no cost to the real crawler, because the real one never sends from those addresses. **Unverifiable** requests should be treated as what they provably are, an ordinary visitor with a name you cannot confirm: apply the same limits and challenges you apply to any other unknown client.

A verified result answers one question only: the request came from the operator's network or key. It does not tell you what the operator will do with the page, whether it respects your robots.txt, or how fast it will crawl. Those are separate measurements, and the [AI crawlers reference](/ai-crawlers) records them per agent where they have been measured.

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-tell-a-real-ai-crawler-from-one-borrowing-its-name%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-tell-a-real-ai-crawler-from-one-borrowing-its-name%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=How%20to%20tell%20a%20real%20AI%20crawler%20from%20one%20borrowing%20its%20name)[Email](mailto:?subject=How%20to%20tell%20a%20real%20AI%20crawler%20from%20one%20borrowing%20its%20name&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-tell-a-real-ai-crawler-from-one-borrowing-its-name)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Can a request fake an AI crawler's user agent?

Yes. The user agent is a header the sender types, so any script on any server can send the exact name of any crawler. The name alone proves nothing about where the request came from. Only a check the operator controls, such as its DNS, its published address list or its signing key, can confirm the identity.

### What is reverse DNS with a forward confirm?

It is a two-step address check. A reverse lookup turns the source address into a hostname, and you confirm the hostname ends in the operator's documented domain. A forward lookup on that hostname must then return the original address. The forward step matters because only the operator can publish address records inside its own domain.

### What if a crawler's operator publishes no address list or hostname?

Then its requests are unverifiable: they can be neither confirmed nor accused. Do not count them as forged, and do not treat them as verified. Handle them like any other unknown visitor, with the same limits and challenges you apply to ordinary traffic.

### How common is a borrowed crawler name?

On 20 November 2026, 11.6% of requests that named a checkable crawler on CrawlCheck's own traffic (24,717 of 213,068 claims) came from outside the operator's published address range. That figure is first-party traffic, not a survey of the web, and the live rate is recomputed on CrawlCheck's forged-traffic page.

### What is a signed crawler request?

A request signed with HTTP Message Signatures (RFC 9421). The crawler signs parts of each request with a private key and publishes the public key in a key directory on its own domain. A valid signature that covers your host and is recent proves the request was made by the key holder. A missing signature proves nothing, because signing is still optional.

### Should I block a crawler by its user agent?

Not on the name alone. A name-based block stops the real crawler and every impostor equally, and a name-based allow lets every impostor in. Verify first: use robots.txt to control verified crawlers, and block or rate limit forged requests at your edge, which costs the real crawler nothing.

## Related findings

[2026-10-02 How to verify Googlebot, GPTBot and other crawlers by IP Verify Googlebot, Bingbot, GPTBot and ClaudeBot by reverse DNS or the operator's IP list. The file for each crawler, the traps, an](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers)[2026-11-13 Claude-User uses two published address lists. We checked one CrawlCheck checked Claude-User against one of two Anthropic address lists and counted real fetches as failed. Fixed 13 November 20](/blog/claude-user-comes-from-two-published-address-lists-we-checked-one)[2026-10-18 How to verify Amazonbot and Amzn-SearchBot by IP address Amazon publishes Amazonbot and Amzn-SearchBot addresses on web pages. How to check a request against them, and how many claims we](/guides/how-to-verify-amazonbot-and-amzn-searchbot)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) · [Listing check](/listing-check) | **Registry:** [congress.gov](/registry/congress.gov) · [flippio.pl](/registry/flippio.pl) · [lhh.com](/registry/lhh.com) · [severadao.ai](/registry/severadao.ai) · [Place: Crestview](/registry/place/fl/crestview) · [app.neilpatel.com](/registry/capabilities/app.neilpatel.com) | **Datasets:** [Technologies: Zendesk](/datasets/technologies/zendesk) · [Problems: ARD legacy path](/datasets/problems/ard-legacy-path)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/how-to-tell-a-real-ai-crawler-from-one-borrowing-its-name.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.