CrawlCheck

Findings · 2026-11-13 · By

14 of 43 crawlers in our catalogue were classed wrong

We counted a block on 43 crawlers against a site. On 13 November 2026 we read each one against its operator's own page. Fourteen were in the wrong class, and nine more were judged by the wrong rule.

Until 13 November 2026, CrawlCheck treated 43 crawlers as search or answer engines, so blocking one counted against a site. Read against each operator's own page, 14 were wrong: Amazonbot is a training crawler, five Google and Bing names are ad or testing tools, two are retired, and six serve one national market. Nine more were judged by a robots.txt rule their operators say may not apply. Every record now carries its source.

A crawler catalogue makes one claim per row: what this crawler is for. Ours decides something that matters to a site owner. If a crawler feeds a search index or an answer engine, blocking it costs visibility, and we say so. If it collects training data, blocking it is a policy choice and we only report it.

On 13 November 2026 we read every crawler in the first group against its operator's own documentation. Of 43, 14 did not belong there.

What was wrong #

GroupCountNamesWhat the operator's page says
Training crawler filed as search1AmazonbotAmazon says what it collects may be used to train its AI models. Amazon search uses a separate crawler
Ad, testing and preview tools5Mediapartners-Google, APIs-Google, Google-InspectionTool, AdsBot-Google-Mobile, BingPreviewAd matching, push notifications, testing tools, ad quality checks and page snapshots. None builds a search index
Retired2msnbot, NeevabotMicrosoft replaced msnbot with bingbot; Neeva closed its search engine in 2023
Single-market search engines6Yandex, YandexBot, Baiduspider, Seznambot, coccocbot-web, PetalBotReal search crawlers, but for one national market. We already reported Naver, Sogou and 360 Search without judging them; these were treated differently for no reason
43 CRAWLERS COUNTED AGAINST A SITE WHEN BLOCKED29 kept15 tools26 single-markettrainingretired14 moved out. Of the 29 kept, 9 are now judged by what the server answers, not by robots.txt.

Nine more, judged by the wrong rule #

Among the 29 that stayed, nine are fetchers that act for a person: ChatGPT-User, Perplexity-User, Amzn-User, Meta-ExternalFetcher, Google-Agent, Google-GeminiNotebook and three with no operator documentation that we could find. We marked a site down when its robots.txt named them. Five operators say on their own pages that such a fetch may ignore robots.txt, so the rule proves little either way. These are now reported in their own row and judged only by what the server actually answers. The operators' statements are set out in robots.txt and AI fetchers: what seven operators say.

One error in the other direction #

The same read found a crawler we had been too lenient about. We had meta-webindexer filed as a training crawler. Meta's page says it exists to improve Meta AI search results, which makes it an index crawler for an answer engine. It is now treated like the others of its kind.

Several unjudged labels were wrong too. Magpie-crawler was described as an AI crawler; it belongs to Brandwatch, a social listening product. VelenPublicWebCrawler is Hunter's. The Awario crawlers do brand monitoring. All were filed under training and are now filed as tools.

Who this affected #

A site that blocked only crawlers from the wrong group was marked down for a choice that cost it nothing. A site that blocked meta-webindexer was not told. When we replayed two large retail robots.txt files through the old and the corrected catalogue, the line about Amazon went from flagged to clean on both. Reports made before 13 November carry the old reading; a new scan carries the corrected one.

Why it happened #

Each class was typed by hand on the day the crawler was added, from whatever was known then, and never read again. Operators rename crawlers, split one into three and retire others. Nothing in our release process compared a class with a source, so an old guess looked exactly like a checked fact.

What changed #

Every search and answer crawler in the catalogue now carries the operator page it was read from, the date it was read, and what that page says a robots.txt rule does. A release that adds one without a source is refused. Where we found no operator page for a name, its page says so instead of implying a source. Retired crawlers have their own group. Changes to the catalogue are published as a dated feed, and each crawler page has a JSON copy with the source in it.

One limit remains. For 21 names we found no operator page on the day we looked, and for those the class still rests on the name and on public reports. Their pages say that.

CrawlCheck’s dataset when this page loaded: 570,959 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

How many crawlers did CrawlCheck class wrongly?

Fourteen of the 43 it treated as search or answer engines on 13 November 2026: one training crawler, five ad, testing and preview tools, two retired crawlers and six single-market search engines. One more, meta-webindexer, was wrong in the other direction.

Is Amazonbot a search crawler or a training crawler?

Amazon's own page says Amazonbot is used to improve its products and services and that what it collects may be used to train Amazon AI models. Amazon search uses a separate crawler, Amzn-SearchBot.

Did this change my site's result?

Only if your robots.txt treated one of the affected crawlers differently from the rest. Reports made before 13 November 2026 carry the old reading; run a new scan for the corrected one.

How does CrawlCheck stop this happening again?

Each search and answer crawler record carries its operator source page and the date it was read, and a release that adds one without a source is refused.

Where can I see what changed in the catalogue?

The catalogue publishes a dated change feed at /ai-crawlers/changes.json, and every crawler page has a JSON copy that includes the operator source.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works