CrawlCheck

Findings · 2026-11-13 · By

Release notes: 7 to 13 November 2026

What changed on CrawlCheck this week: ready-made block rules, operator address lists in one file, a corrected crawler catalogue, and four fixes to our own measurements.

Between 7 and 13 November 2026 CrawlCheck added ready-made robots.txt and server block rules, a single file mirroring 18 operator address lists, a change feed for the crawler catalogue and a Markdown copy announced on every page. It corrected the class of 14 crawlers, accepted Claude-User from both of Anthropic's published lists, stopped its SDKs cutting tool lists at 300, and closed a mirror that had inflated its own forgery counts.

What shipped, and where to find it
Ready-made policies
What
Ready-made robots.txt files and matching block rules for nginx, Apache, Caddy and Cloudflare, in three sets
Since
13 Nov
ip-ranges.json
What
18 operator address lists mirrored in one file, refreshed hourly
Since
13 Nov
changes.json
What
A dated change feed for the crawler catalogue: send your last position, get only what is new
Since
13 Nov
Any page, add .md
What
Every indexable page names its Markdown copy, translated pages included
Since
13 Nov
Crawler catalogue
What
Crawler comparison pages say what separates the two crawlers
Since
13 Nov
SDK
What
A digest of an MCP server's whole tool list in both SDKs
Since
13 Nov

One week, in order of how many people it affects. Every item links to the page or the write-up behind it.

New this week #

WhatWhereSince
Ready-made robots.txt files and matching block rules for nginx, Apache, Caddy and Cloudflare, in three setsReady-made policies13 Nov
18 operator address lists mirrored in one file, refreshed hourlyip-ranges.json13 Nov
A dated change feed for the crawler catalogue: send your last position, get only what is newchanges.json13 Nov
Every indexable page names its Markdown copy, translated pages includedAny page, add .md13 Nov
Crawler comparison pages say what separates the two crawlersCrawler catalogue13 Nov
A digest of an MCP server's whole tool list in both SDKsSDK13 Nov

Corrections to our own measurements #

Four things we had wrong and fixed. Each has its own account.

The crawler catalogue. 14 of 43 crawlers were classed wrong. The catalogue now lists 116 crawlers and every record carries the operator page it was read from. Three Google ad and API crawlers are now treated as ignoring the wildcard group in robots.txt, as Google's own page says they do.

Forgery counts. A mirror we ran relayed real crawlers to this site and we counted them as failed checks. The mirror is closed and counts before 13 November carry a note.

Claude-User. We checked one of two published address lists. Both are accepted now.

Tool lists. The lockfile and both SDKs stopped at 300 tools without saying so. Fixed on 11 November in @crawlcheck/sdk 1.14.0 and crawlcheck 1.15.0 on PyPI.

Smaller changes #

Decisions and log heads have been signed with their own keys since 9 November; the keys page lists which key signs what. The index alert no longer counts tracked booking links as missing pages. Seven new guides and findings went up on 13 November, all in the library.

If you use the SDKs #

Upgrade to the current release. If you pinned a lockfile for a server with more than 300 tools before 11 November, create it again.

CrawlCheck’s dataset when this page loaded: 570,959 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

What did CrawlCheck release between 7 and 13 November 2026?

Ready-made robots.txt and server block rules, one file mirroring 18 operator address lists, a change feed for the crawler catalogue, Markdown copies announced on every page, and a tool-list digest in both SDKs.

What did CrawlCheck correct that week?

The class of 14 crawlers in its catalogue, forgery counts inflated by its own mirror, Claude-User verification against a second published address list, and SDKs that cut tool lists at 300.

Do I need to do anything?

Only if you use the SDKs: upgrade to the current release, and recreate any lockfile made before 11 November 2026 for a server with more than 300 tools.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works