Findings · 2026-11-13 · By VSNARY | Emmanuel Orta
14 of 43 crawlers in our catalogue were classed wrong
We counted a block on 43 crawlers against a site. On 13 November 2026 we read each one against its operator's own page. Fourteen were in the wrong class, and nine more were judged by the wrong rule.
Until 13 November 2026, CrawlCheck treated 43 crawlers as search or answer engines, so blocking one counted against a site. Read against each operator's own page, 14 were wrong: Amazonbot is a training crawler, five Google and Bing names are ad or testing tools, two are retired, and six serve one national market. Nine more were judged by a robots.txt rule their operators say may not apply. Every record now carries its source.
A crawler catalogue makes one claim per row: what this crawler is for. Ours decides something that matters to a site owner. If a crawler feeds a search index or an answer engine, blocking it costs visibility, and we say so. If it collects training data, blocking it is a policy choice and we only report it.
On 13 November 2026 we read every crawler in the first group against its operator's own documentation. Of 43, 14 did not belong there.
What was wrong #
| Group | Count | Names | What the operator's page says |
|---|---|---|---|
| Training crawler filed as search | 1 | Amazonbot | Amazon says what it collects may be used to train its AI models. Amazon search uses a separate crawler |
| Ad, testing and preview tools | 5 | Mediapartners-Google, APIs-Google, Google-InspectionTool, AdsBot-Google-Mobile, BingPreview | Ad matching, push notifications, testing tools, ad quality checks and page snapshots. None builds a search index |
| Retired | 2 | msnbot, Neevabot | Microsoft replaced msnbot with bingbot; Neeva closed its search engine in 2023 |
| Single-market search engines | 6 | Yandex, YandexBot, Baiduspider, Seznambot, coccocbot-web, PetalBot | Real search crawlers, but for one national market. We already reported Naver, Sogou and 360 Search without judging them; these were treated differently for no reason |
Nine more, judged by the wrong rule #
Among the 29 that stayed, nine are fetchers that act for a person: ChatGPT-User, Perplexity-User, Amzn-User, Meta-ExternalFetcher, Google-Agent, Google-GeminiNotebook and three with no operator documentation that we could find. We marked a site down when its robots.txt named them. Five operators say on their own pages that such a fetch may ignore robots.txt, so the rule proves little either way. These are now reported in their own row and judged only by what the server actually answers. The operators' statements are set out in robots.txt and AI fetchers: what seven operators say.
One error in the other direction #
The same read found a crawler we had been too lenient about. We had meta-webindexer filed as a training crawler. Meta's page says it exists to improve Meta AI search results, which makes it an index crawler for an answer engine. It is now treated like the others of its kind.
Several unjudged labels were wrong too. Magpie-crawler was described as an AI crawler; it belongs to Brandwatch, a social listening product. VelenPublicWebCrawler is Hunter's. The Awario crawlers do brand monitoring. All were filed under training and are now filed as tools.
Who this affected #
A site that blocked only crawlers from the wrong group was marked down for a choice that cost it nothing. A site that blocked meta-webindexer was not told. When we replayed two large retail robots.txt files through the old and the corrected catalogue, the line about Amazon went from flagged to clean on both. Reports made before 13 November carry the old reading; a new scan carries the corrected one.
Why it happened #
Each class was typed by hand on the day the crawler was added, from whatever was known then, and never read again. Operators rename crawlers, split one into three and retire others. Nothing in our release process compared a class with a source, so an old guess looked exactly like a checked fact.
What changed #
Every search and answer crawler in the catalogue now carries the operator page it was read from, the date it was read, and what that page says a robots.txt rule does. A release that adds one without a source is refused. Where we found no operator page for a name, its page says so instead of implying a source. Retired crawlers have their own group. Changes to the catalogue are published as a dated feed, and each crawler page has a JSON copy with the source in it.
One limit remains. For 21 names we found no operator page on the day we looked, and for those the class still rests on the name and on public reports. Their pages say that.
CrawlCheck’s dataset when this page loaded: 570,959 sites measured.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
How many crawlers did CrawlCheck class wrongly?
Fourteen of the 43 it treated as search or answer engines on 13 November 2026: one training crawler, five ad, testing and preview tools, two retired crawlers and six single-market search engines. One more, meta-webindexer, was wrong in the other direction.
Is Amazonbot a search crawler or a training crawler?
Amazon's own page says Amazonbot is used to improve its products and services and that what it collects may be used to train Amazon AI models. Amazon search uses a separate crawler, Amzn-SearchBot.
Did this change my site's result?
Only if your robots.txt treated one of the affected crawlers differently from the rest. Reports made before 13 November 2026 carry the old reading; run a new scan for the corrected one.
How does CrawlCheck stop this happening again?
Each search and answer crawler record carries its operator source page and the date it was read, and a release that adds one without a source is refused.
Where can I see what changed in the catalogue?
The catalogue publishes a dated change feed at /ai-crawlers/changes.json, and every crawler page has a JSON copy that includes the operator source.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.