Agent view · Findings · 2026-11-13
14 of 43 crawlers in our catalogue were classed wrong
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/fourteen-of-43-crawlers-in-our-catalogue-were-classed-wrong: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/fourteen-of-43-crawlers-in-our-catalogue-were-classed-wrong>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 13,460 |
| words | 1,864 |
| sha-256 | 4342c941a66c4552732a9d778179416b9ff8bd4f8b24b4a56656545d51afa03e |
| repr-digest | sha-256=:Q0LJQaZsRVJzKp13gXlBa5/4vU+LJLSlZlZUXVGvoD4=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795180565;expires=1795184165;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="_7cDt-wsVGL52QLRPTZGYw";tag="crawlcheck-page" |
| signature | sig1=:5ojf3mEix+jxBkk+2HVUj6YCje4zBSQeabBFMALylul2Ust20sMK6RB6Qod1hwthOjFqlKB9xM3sVsnB2yHFCw==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Findings · 2026-11-13 · By [VSNARY | Emmanuel Orta](/author) # 14 of 43 crawlers in our catalogue were classed wrong We counted a block on 43 crawlers against a site. On 13 November 2026 we read each one against its operator's own page. Fourteen were in the wrong class, and nine more were judged by the wrong rule. Until 13 November 2026, CrawlCheck treated 43 crawlers as search or answer engines, so blocking one counted against a site. Read against each operator's own page, 14 were wrong: Amazonbot is a training crawler, five Google and Bing names are ad or testing tools, two are retired, and six serve one national market. Nine more were judged by a robots.txt rule their operators say may not apply. Every record now carries its source. On this page - What was wrong - Nine more, judged by the wrong rule - One error in the other direction - Who this affected - Why it happened - What changed A crawler catalogue makes one claim per row: what this crawler is for. Ours decides something that matters to a site owner. If a crawler feeds a search index or an [answer engine](/glossary/answer-engine), blocking it costs visibility, and we say so. If it collects [training data](/glossary/training-data), blocking it is a policy choice and we only report it. On 13 November 2026 we read every crawler in the first group against its operator's own documentation. Of 43, 14 did not belong there. ## What was wrong # | Group | Count | Names | What the operator's page says | | --- | --- | --- | --- | | Training crawler filed as search | 1 | Amazonbot | Amazon says what it collects may be used to train its AI models. Amazon search uses a separate crawler | | Ad, testing and preview tools | 5 | Mediapartners-Google, APIs-Google, Google-InspectionTool, AdsBot-Google-Mobile, BingPreview | Ad matching, push notifications, testing tools, ad quality checks and page snapshots. None builds a search index | | Retired | 2 | msnbot, Neevabot | Microsoft replaced msnbot with bingbot; Neeva closed its search engine in 2023 | | Single-market search engines | 6 | Yandex, YandexBot, Baiduspider, Seznambot, coccocbot-web, PetalBot | Real search crawlers, but for one national market. We already reported Naver, Sogou and 360 Search without judging them; these were treated differently for no reason | ## Nine more, judged by the wrong rule # Among the 29 that stayed, nine are fetchers that act for a person: ChatGPT-User, Perplexity-User, Amzn-User, Meta-ExternalFetcher, [Google-Agent](/glossary/google-agent), Google-GeminiNotebook and three with no operator documentation that we could find. We marked a site down when its [robots.txt](/glossary/robots-txt) named them. Five operators say on their own pages that such a fetch may ignore robots.txt, so the rule proves little either way. These are now reported in their own row and judged only by what the server actually answers. The operators' statements are set out in [robots.txt and AI fetchers: what seven operators say](/guides/robots-txt-does-not-stop-user-triggered-ai-fetchers). ## One error in the other direction # The same read found a crawler we had been too lenient about. We had meta-webindexer filed as a training crawler. [Meta's page](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) says it exists to improve Meta AI search results, which makes it an index crawler for an answer engine. It is now treated like the others of its kind. Several unjudged labels were wrong too. Magpie-crawler was described as an AI crawler; it belongs to Brandwatch, a social listening product. VelenPublicWebCrawler is Hunter's. The Awario crawlers do brand monitoring. All were filed under training and are now filed as tools. ## Who this affected # A site that blocked only crawlers from the wrong group was marked down for a choice that cost it nothing. A site that blocked meta-webindexer was not told. When we replayed two large retail robots.txt files through the old and the corrected catalogue, the line about Amazon went from flagged to clean on both. Reports made before 13 November carry the old reading; a new scan carries the corrected one. ## Why it happened # Each class was typed by hand on the day the crawler was added, from whatever was known then, and never read again. Operators rename crawlers, split one into three and retire others. Nothing in our release process compared a class with a source, so an old guess looked exactly like a checked fact. ## What changed # Every search and answer crawler in the [catalogue](/ai-crawlers) now carries the operator page it was read from, the date it was read, and what that page says a robots.txt rule does. A release that adds one without a source is refused. Where we found no operator page for a name, its page says so instead of implying a source. Retired crawlers have their own group. Changes to the catalogue are published as a [dated feed](/ai-crawlers/changes.json), and each crawler page has a JSON copy with the source in it. One limit remains. For 21 names we found no operator page on the day we looked, and for those the class still rests on the name and on public reports. Their pages say that. CrawlCheck’s dataset when this page loaded: 570,959 sites measured. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Ffourteen-of-43-crawlers-in-our-catalogue-were-classed-wrong%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Ffourteen-of-43-crawlers-in-our-catalogue-were-classed-wrong%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=14%20of%2043%20crawlers%20in%20our%20catalogue%20were%20classed%20wrong)[Email](mailto:?subject=14%20of%2043%20crawlers%20in%20our%20catalogue%20were%20classed%20wrong&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Ffourteen-of-43-crawlers-in-our-catalogue-were-classed-wrong)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### How many crawlers did CrawlCheck class wrongly? Fourteen of the 43 it treated as search or answer engines on 13 November 2026: one training crawler, five ad, testing and preview tools, two retired crawlers and six single-market search engines. One more, meta-webindexer, was wrong in the other direction. ### Is Amazonbot a search crawler or a training crawler? Amazon's own page says Amazonbot is used to improve its products and services and that what it collects may be used to train Amazon AI models. Amazon search uses a separate crawler, Amzn-SearchBot. ### Did this change my site's result? Only if your robots.txt treated one of the affected crawlers differently from the rest. Reports made before 13 November 2026 carry the old reading; run a new scan for the corrected one. ### How does CrawlCheck stop this happening again? Each search and answer crawler record carries its operator source page and the date it was read, and a release that adds one without a source is refused. ### Where can I see what changed in the catalogue? The catalogue publishes a dated change feed at /ai-crawlers/changes.json, and every crawler page has a JSON copy that includes the operator source. ## Related findings [2026-11-13 Release notes: 7 to 13 November 2026 CrawlCheck release notes for 7 to 13 November 2026: block rules for four servers, crawler IP ranges in one file, catalogue correct](/blog/release-notes-7-to-13-november-2026)[2026-09-01 Seven site types, and the report checks we corrected We ran seven very different sites through every graded check and found some failing sites for non-defects. Which checks changed, a](/blog/we-read-every-scored-row-against-seven-sites-here-is-what-we-unscored)[2026-10-08 AI agents acting for people: 613 visits, one read robots.txt ChatGPT-User made 526 verified visits, Claude-User 79, Perplexity-User 8. Only Claude-User read robots.txt; none fetched a disallo](/blog/agents-acting-for-people-613-verified-visits-one-read-robots-txt) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Pages:** [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [buickgmcbrooksville.com](/registry/buickgmcbrooksville.com) · [drom.ru](/registry/drom.ru) · [jonhallchevrolet.com](/registry/jonhallchevrolet.com) · [plombier-paris.fr](/registry/plombier-paris.fr) · [ya.ru](/registry/ya.ru) | **Glossary:** [C2PA claim](/glossary/c2pa-claim) · [Dataset (schema.org)](/glossary/dataset-schema) | **Directory:** [Machinerelations](/directory/machinerelations) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /blog/fourteen-of-43-crawlers-in-our-catalogue-were-classed-wrong.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.