Agent view · Findings · 2026-11-13
Release notes: 7 to 13 November 2026
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/release-notes-7-to-13-november-2026: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/release-notes-7-to-13-november-2026>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 11,926 |
| words | 1,576 |
| sha-256 | 65f0ad99d726e695657a450d6a8ee3d11f48a252effbc40e0ebbbd97186b3e87 |
| repr-digest | sha-256=:ZfCtmdcm5pVlekUNao7j0R9IolLv+8QODru9lxhrPoc=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795180559;expires=1795184159;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="NA3DPZeOH6BX-DNTzX9qvQ";tag="crawlcheck-page" |
| signature | sig1=:wWcelVKyQkiPzUQfdvGBmXoT3lYUtooSYf8s8XCr7ZuXHOHK5b6rQIVt1Eg5KLuOgWLi4kqY7RxWKkTb7BcDAg==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Findings · 2026-11-13 · By [VSNARY | Emmanuel Orta](/author) # Release notes: 7 to 13 November 2026 What changed on CrawlCheck this week: ready-made block rules, operator address lists in one file, a corrected crawler catalogue, and four fixes to our own measurements. Between 7 and 13 November 2026 CrawlCheck added ready-made robots.txt and server block rules, a single file mirroring 18 operator address lists, a change feed for the crawler catalogue and a Markdown copy announced on every page. It corrected the class of 14 crawlers, accepted Claude-User from both of Anthropic's published lists, stopped its SDKs cutting tool lists at 300, and closed a mirror that had inflated its own forgery counts. On this page - New this week - Corrections to our own measurements - Smaller changes - If you use the SDKs What shipped, and where to find it Ready-made policies What Ready-made robots.txt files and matching block rules for nginx, Apache, Caddy and Cloudflare, in three sets Since 13 Nov ip-ranges.json What 18 operator address lists mirrored in one file, refreshed hourly Since 13 Nov changes.json What A dated change feed for the crawler catalogue: send your last position, get only what is new Since 13 Nov Any page, add .md What Every indexable page names its Markdown copy, translated pages included Since 13 Nov Crawler catalogue What Crawler comparison pages say what separates the two crawlers Since 13 Nov SDK What A digest of an MCP server's whole tool list in both SDKs Since 13 Nov One week, in order of how many people it affects. Every item links to the page or the write-up behind it. ## New this week # | What | Where | Since | | --- | --- | --- | | Ready-made [robots.txt](/glossary/robots-txt) files and matching block rules for nginx, Apache, Caddy and Cloudflare, in three sets | [Ready-made policies](/robots-policies) | 13 Nov | | 18 operator address lists mirrored in one file, refreshed hourly | [ip-ranges.json](/ai-crawlers/ip-ranges.json) | 13 Nov | | A dated change feed for the crawler catalogue: send your last position, get only what is new | [changes.json](/ai-crawlers/changes.json) | 13 Nov | | Every indexable page names its Markdown copy, translated pages included | Any page, add .md | 13 Nov | | Crawler comparison pages say what separates the two crawlers | [Crawler catalogue](/ai-crawlers) | 13 Nov | | A digest of an [MCP server](/glossary/mcp-server)'s whole tool list in both SDKs | [SDK](/sdk) | 13 Nov | ## Corrections to our own measurements # Four things we had wrong and fixed. Each has its own account. **The crawler catalogue.** [14 of 43 crawlers were classed wrong](/blog/fourteen-of-43-crawlers-in-our-catalogue-were-classed-wrong). The catalogue now lists 116 crawlers and every record carries the operator page it was read from. Three Google ad and API crawlers are now treated as ignoring the wildcard group in robots.txt, as Google's own page says they do. **Forgery counts.** [A mirror we ran relayed real crawlers to this site](/blog/our-mirror-relayed-real-crawlers-and-inflated-our-forgery-counts) and we counted them as failed checks. The mirror is closed and counts before 13 November carry a note. **Claude-User.** [We checked one of two published address lists](/blog/claude-user-comes-from-two-published-address-lists-we-checked-one). Both are accepted now. **Tool lists.** [The lockfile and both SDKs stopped at 300 tools without saying so](/blog/our-sdks-cut-a-tool-list-at-300-with-no-warning). Fixed on 11 November in @crawlcheck/sdk 1.14.0 and crawlcheck 1.15.0 on PyPI. ## Smaller changes # Decisions and log heads have been signed with their own keys since 9 November; the [keys page](/keys) lists which key signs what. The index alert no longer counts [tracked booking links as missing pages](/blog/booking-links-counted-as-missing-sitemap-pages). Seven new guides and findings went up on 13 November, all in the [library](/blog). ## If you use the SDKs # Upgrade to the current release. If you pinned a lockfile for a server with more than 300 tools before 11 November, create it again. CrawlCheck’s dataset when this page loaded: 570,959 sites measured. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Frelease-notes-7-to-13-november-2026%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Frelease-notes-7-to-13-november-2026%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Release%20notes%3A%207%20to%2013%20November%202026)[Email](mailto:?subject=Release%20notes%3A%207%20to%2013%20November%202026&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Frelease-notes-7-to-13-november-2026)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### What did CrawlCheck release between 7 and 13 November 2026? Ready-made robots.txt and server block rules, one file mirroring 18 operator address lists, a change feed for the crawler catalogue, Markdown copies announced on every page, and a tool-list digest in both SDKs. ### What did CrawlCheck correct that week? The class of 14 crawlers in its catalogue, forgery counts inflated by its own mirror, Claude-User verification against a second published address list, and SDKs that cut tool lists at 300. ### Do I need to do anything? Only if you use the SDKs: upgrade to the current release, and recreate any lockfile made before 11 November 2026 for a server with more than 300 tools. ## Related findings [2026-11-13 14 of 43 crawlers in our catalogue were classed wrong CrawlCheck classed 14 of 43 crawlers wrongly until 13 November 2026, including Amazonbot, msnbot and Yandex. What was wrong and wh](/blog/fourteen-of-43-crawlers-in-our-catalogue-were-classed-wrong)[2026-11-13 AI crawler IP ranges from 18 operator lists, in one file One JSON file mirrors 18 operator-published crawler address lists, refreshed hourly, with a checksum and status per source.](/blog/ai-crawler-ip-ranges-from-18-operator-lists-in-one-file)[2026-11-13 Claude-User uses two published address lists. We checked one CrawlCheck checked Claude-User against one of two Anthropic address lists and counted real fetches as failed. Fixed 13 November 20](/blog/claude-user-comes-from-two-published-address-lists-we-checked-one) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Pages:** [/](/) · [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) · [Policy as code](/docs/policy-as-code) | **Registry:** [ya.ru](/registry/ya.ru) · [centralbuickgmctruck.com](/registry/capabilities/centralbuickgmctruck.com) | **Datasets:** [Hawaii](/datasets/state/hawaii) · [Technologies: Webflow](/datasets/technologies/webflow) | **Guides:** [Content signals in robots txt search AI input AI train](/guides/content-signals-in-robots-txt-search-ai-input-ai-train) | **Glossary:** [AI classifiers: Answer generation](/glossary/ai-classifiers/answer-generation) · [Layout-aware parsing](/glossary/layout-aware-parsing) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /blog/release-notes-7-to-13-november-2026.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.