CrawlCheck

Agent view · Findings · 2026-11-13

Our mirror relayed real crawlers and inflated forgery counts

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/our-mirror-relayed-real-crawlers-and-inflated-our-forgery-counts: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/our-mirror-relayed-real-crawlers-and-inflated-our-forgery-counts>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes12,380
words1,709
sha-256a75307bdd9c864b9341805cfeb7010669a1de84dd8967ba859789f07df3d3426
repr-digestsha-256=:p1MHvdnIZLk0GAXP63AQZpod6E3YlnuoWXifB989NCY=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795181429;expires=1795185029;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="9KG28_UogGeLqMGwToKktA";tag="crawlcheck-page"
signaturesig1=:fNht7E3VF3/uB92s+tr/WCQfHewjjBAStxiknNb2xQwO5ytPjBSft3TUYvOVqzqXCERSJvIBWsdxikf+v15IBg==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Findings · 2026-11-13 · By [VSNARY | Emmanuel Orta](/author)

# Our mirror relayed real crawlers and inflated forgery counts

A copy of this site that we ran ourselves passed crawler requests back to us under the crawler's own name. Our verifier counted them as failed checks. The counts were ours to get right.

Until 13 November 2026 a mirror of crawlcheck.io that we ran ourselves relayed crawler requests to this site from a CDN address, under each crawler's own name. Our verifier compares the address with the operator's published list, so relayed requests counted as failed or uncheckable. That covers 68,801 hits recorded under Meta-ExternalAgent and part of Amazonbot's failed count. The mirror is closed. The earlier counts cannot be separated and should not be quoted as a forgery rate.

On this page

- What happened
- How that became a wrong number
- How much is affected
- What we changed
- Check your own setup

What reached this site, what we recorded, what was true

A request named Amazonbot from a CDN address

What we recorded

Failed the address check

What was true

A real Amazonbot fetch of the mirror, passed on by us

A request named Meta-ExternalAgent from a CDN address

What we recorded

A hit that could not be verified

What was true

A real Meta-ExternalAgent fetch of the mirror, passed on by us

We publish how many requests arrive here under each crawler's name, and how many of those come from an address the operator lists. A request that names a crawler but comes from somewhere else is counted as failing the check. On 13 November 2026 we found that a large share of those failures were our own doing.

## What happened #

We had built a mirror of this site on three other hostnames, as an outside [vantage point](/glossary/vantage-point) for our own tests. It served every page and passed every request through to crawlcheck.io. It carried our normal [robots.txt](/glossary/robots-txt), which allows crawlers. Nothing of ours still read it, but it was never switched off.

Two crawlers found it and crawled it: Meta-ExternalAgent and Amazonbot. Each of their requests to the mirror became a request to crawlcheck.io, sent by the mirror from a CDN address, with the crawler's [user agent](/glossary/user-agent) still attached.

## How that became a wrong number #

| What reached this site | What we recorded | What was true |
| --- | --- | --- |
| A request named Amazonbot from a CDN address | Failed the address check | A real Amazonbot fetch of the mirror, passed on by us |
| A request named Meta-ExternalAgent from a CDN address | A hit that could not be verified | A real Meta-ExternalAgent fetch of the mirror, passed on by us |

The check itself worked as designed. The address was not on the operator's list, because the address was ours. The error was upstream of the check: we were the ones sending the traffic.

## How much is affected #

The 68,801 hits recorded under Meta-ExternalAgent trace to the mirror. For Amazonbot, and to a smaller degree other crawlers that publish addresses, part of the failed count is relayed traffic. We cannot say exactly how much, because a relayed request and a forged one look the same in what we stored.

So we are not restating the figures. The [telemetry data](/api/telemetry) now carries a note saying that counts recorded before 13 November 2026 16:04 UTC include relayed traffic and should not be quoted as a forgery rate. Counts after that time do not include it.

## What we changed #

The mirror now answers 410 on every path, sends nothing onward, and its robots.txt disallows everything. We checked all three hostnames by hand, with GET and POST.

## Check your own setup #

This is not special to us. Any reverse proxy, staging copy or mirror that forwards requests to your main site keeps the visitor's user agent and replaces the visitor's address with its own. If you judge crawlers by address, as [you should](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers), those requests will look forged.

Search your logs for crawler user agents arriving from your own servers or your own [CDN](/glossary/cdn) account. If you find any, something you run is relaying them. Either close it to crawlers or pass the original address through and check that one.

CrawlCheck’s dataset when this page loaded: 570,959 sites measured.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-mirror-relayed-real-crawlers-and-inflated-our-forgery-counts%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-mirror-relayed-real-crawlers-and-inflated-our-forgery-counts%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Our%20mirror%20relayed%20real%20crawlers%20and%20inflated%20forgery%20counts)[Email](mailto:?subject=Our%20mirror%20relayed%20real%20crawlers%20and%20inflated%20forgery%20counts&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-mirror-relayed-real-crawlers-and-inflated-our-forgery-counts)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Did Meta or Amazon do anything wrong here?

No. Their crawlers fetched a public copy of our site that allowed crawling. We relayed those fetches to our main site ourselves, and our own check then counted them as failures.

### Which CrawlCheck figures are affected?

Per-crawler counts recorded before 13 November 2026 16:04 UTC: most hits under Meta-ExternalAgent, and part of the failed-verification counts for crawlers that publish addresses, Amazonbot above all.

### Can the old counts be corrected?

Not exactly. A relayed request and a forged request look the same in the stored records, so the data carries a note instead of a restated figure.

### How do I know if my own proxy is relaying crawlers?

Look in your main site's logs for crawler user agents whose source address belongs to your own servers or CDN account. A real crawler never arrives from your own infrastructure.

## Related findings

[2026-11-13 Claude-User uses two published address lists. We checked one CrawlCheck checked Claude-User against one of two Anthropic address lists and counted real fetches as failed. Fixed 13 November 20](/blog/claude-user-comes-from-two-published-address-lists-we-checked-one)[2026-11-13 Block AI crawlers in nginx, Apache and Caddy (tested) Paste-ready nginx, Apache and Caddy rules that return 403 to AI crawlers, tested on real servers on 13 November 2026, with the pit](/guides/block-ai-crawlers-nginx-apache-caddy)[2026-10-07 Eight weeks of AI crawlers on one site: who was really who 24,627 verified crawler requests to crawlcheck.io in eight weeks: who was real, which names only came forged, and what verified ag](/blog/eight-weeks-of-agent-conduct-who-was-really-who)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) | **Datasets:** [Technologies: Tailwind CSS](/datasets/technologies/tailwind-css) · [Problems: LLMs link broken](/datasets/problems/llms-link-broken) | **Blog:** [AI agents cannot refuse what they cannot see: 27 traps](/blog/ai-agents-cannot-refuse-what-they-cannot-see) | **AI crawlers:** [Blexbot](/ai-crawlers/blexbot) | **Glossary:** [AI classifiers: Temporal analysis](/glossary/ai-classifiers/temporal-analysis) · [Corpus](/glossary/corpus) · [DNS timeout](/glossary/dns-timeout) · [Broken link](/glossary/broken-link)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /blog/our-mirror-relayed-real-crawlers-and-inflated-our-forgery-counts.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.