Findings · 2026-11-13 · By VSNARY | Emmanuel Orta
Our mirror relayed real crawlers and inflated forgery counts
A copy of this site that we ran ourselves passed crawler requests back to us under the crawler's own name. Our verifier counted them as failed checks. The counts were ours to get right.
Until 13 November 2026 a mirror of crawlcheck.io that we ran ourselves relayed crawler requests to this site from a CDN address, under each crawler's own name. Our verifier compares the address with the operator's published list, so relayed requests counted as failed or uncheckable. That covers 68,801 hits recorded under Meta-ExternalAgent and part of Amazonbot's failed count. The mirror is closed. The earlier counts cannot be separated and should not be quoted as a forgery rate.
We publish how many requests arrive here under each crawler's name, and how many of those come from an address the operator lists. A request that names a crawler but comes from somewhere else is counted as failing the check. On 13 November 2026 we found that a large share of those failures were our own doing.
What happened #
We had built a mirror of this site on three other hostnames, as an outside vantage point for our own tests. It served every page and passed every request through to crawlcheck.io. It carried our normal robots.txt, which allows crawlers. Nothing of ours still read it, but it was never switched off.
Two crawlers found it and crawled it: Meta-ExternalAgent and Amazonbot. Each of their requests to the mirror became a request to crawlcheck.io, sent by the mirror from a CDN address, with the crawler's user agent still attached.
How that became a wrong number #
| What reached this site | What we recorded | What was true |
|---|---|---|
| A request named Amazonbot from a CDN address | Failed the address check | A real Amazonbot fetch of the mirror, passed on by us |
| A request named Meta-ExternalAgent from a CDN address | A hit that could not be verified | A real Meta-ExternalAgent fetch of the mirror, passed on by us |
The check itself worked as designed. The address was not on the operator's list, because the address was ours. The error was upstream of the check: we were the ones sending the traffic.
How much is affected #
The 68,801 hits recorded under Meta-ExternalAgent trace to the mirror. For Amazonbot, and to a smaller degree other crawlers that publish addresses, part of the failed count is relayed traffic. We cannot say exactly how much, because a relayed request and a forged one look the same in what we stored.
So we are not restating the figures. The telemetry data now carries a note saying that counts recorded before 13 November 2026 16:04 UTC include relayed traffic and should not be quoted as a forgery rate. Counts after that time do not include it.
What we changed #
The mirror now answers 410 on every path, sends nothing onward, and its robots.txt disallows everything. We checked all three hostnames by hand, with GET and POST.
Check your own setup #
This is not special to us. Any reverse proxy, staging copy or mirror that forwards requests to your main site keeps the visitor's user agent and replaces the visitor's address with its own. If you judge crawlers by address, as you should, those requests will look forged.
Search your logs for crawler user agents arriving from your own servers or your own CDN account. If you find any, something you run is relaying them. Either close it to crawlers or pass the original address through and check that one.
CrawlCheck’s dataset when this page loaded: 570,959 sites measured.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
Did Meta or Amazon do anything wrong here?
No. Their crawlers fetched a public copy of our site that allowed crawling. We relayed those fetches to our main site ourselves, and our own check then counted them as failures.
Which CrawlCheck figures are affected?
Per-crawler counts recorded before 13 November 2026 16:04 UTC: most hits under Meta-ExternalAgent, and part of the failed-verification counts for crawlers that publish addresses, Amazonbot above all.
Can the old counts be corrected?
Not exactly. A relayed request and a forged request look the same in the stored records, so the data carries a note instead of a restated figure.
How do I know if my own proxy is relaying crawlers?
Look in your main site's logs for crawler user agents whose source address belongs to your own servers or CDN account. A real crawler never arrives from your own infrastructure.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.