CrawlCheck

Agent view · Findings · 2026-10-03

Release notes 3 Oct 2026: Resolve, census, rulebook

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/release-notes-3-october-2026-resolve-protocol-census-rulebook: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/release-notes-3-october-2026-resolve-protocol-census-rulebook>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes15,232
words2,163
sha-256620c26ea481787c8021bf1ea9c0852b9d959d7230edbf7d11db1a2c0c70f2c4b
repr-digestsha-256=:Ygwm6kgXh8gCG/HqnAhSudlZ1yMO2/fRHbGiwMcPLEs=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311487;expires=1795315087;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="fk1hG8gItDCfygiv3eZ_Jg";tag="crawlcheck-page"
signaturesig1=:BsCPXVxYtoedqVPiHZJqAHd6gjVP2uBh2YzyO8kYaliEZamC9JvFaEJC9+fkSwAOxlW/OJKM86l6ItMmW66cBA==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Findings · 2026-10-03 · By [VSNARY | Emmanuel Orta](/author)

# Release notes 3 Oct 2026: Resolve, census, rulebook

What changed on crawlcheck.io between 29 September and 3 October 2026: a signed answer AI agents can read before acting, the first agentic-web census and a correction to it, a public rulebook, receipts, disputes and independent observers.

Between 29 September and 3 October 2026, crawlcheck.io gained Resolve, one signed answer an AI agent can read about a domain before acting on it, with no overall score. The first agentic-web census measured 991 sites from the Tranco top 1,000, and its first write-up was corrected the same day. Every finding CrawlCheck can raise is now listed in a public rulebook, a fix ends in a signed before-and-after receipt, and anyone can dispute a finding in a public ledger.

On this page

- What is Resolve?
- The census, and the correction to it
- Rules, receipts and disputes in public
- Second and third networks
- Developers, privacy and security
- Pricing and the site

What changed, and what it means for you

Resolve

What it means for you

An agent reads one signed answer about a domain before it fetches, quotes or acts on it

Agentic-web census

What it means for you

The same frozen list of top sites measured repeatedly, so figures can be compared run to run

Public rulebook

What it means for you

Every finding CrawlCheck can raise, with what it means, how serious it is and how to fix it

Signed receipts

What it means for you

A fix ends in a before-and-after record anyone can verify offline

Dispute ledger

What it means for you

Anyone can challenge a finding; the verdict is published and signed

Independent observers

What it means for you

Readings from separate networks; disagreements are shown, never merged

Export or delete your data

What it means for you

A site owner can take or remove everything held about their domain

Evidence and Intelligence

What it means for you

Two new paid lines; nothing that was free moved behind them

Second note in the Releases series. Every line names something that shipped; nothing here is planned. The previous note is [29 September](/blog/release-notes-29-september-2026-finding-states-sitemap-index-trust-center).

| Change | What it means for you |
| --- | --- |
| Resolve | An agent reads one signed answer about a domain before it fetches, quotes or acts on it |
| Agentic-web census | The same frozen list of top sites measured repeatedly, so figures can be compared run to run |
| Public rulebook | Every finding CrawlCheck can raise, with what it means, how serious it is and how to fix it |
| Signed receipts | A fix ends in a before-and-after record anyone can verify offline |
| Dispute ledger | Anyone can challenge a finding; the verdict is published and signed |
| Independent observers | Readings from separate networks; disagreements are shown, never merged |
| Export or delete your data | A site owner can take or remove everything held about their domain |
| Evidence and Intelligence | Two new paid lines; nothing that was free moved behind them |

## What is Resolve? #

Resolve answers one question for an [AI agent](/glossary/ai-agent): what should it know about this domain before acting? The answer covers the crawl policy the site declares, what each crawler identity was actually served, the machine files, who the site says it is and whether outside sources confirm it, and any open findings. There is deliberately no overall score; an agent decides on the section its action depends on. Every answer is signed with a published key and can be checked without contacting this site. A domain that has never been measured says so and is queued; an opted-out domain says only that.

## The census, and the correction to it #

The first census froze one Tranco top-1,000 list so every run measures the same sites: 991 members after nine excluded hosts, counted and never named. Each site is fetched as a browser and as the main AI crawlers, and its [robots.txt](/glossary/robots-txt) is read for 12 AI agents per RFC 9309.

**Correction, 3 October:** the first version of this note said the homepages that did not answer with a page were walling off datacenter traffic. The records said otherwise. 287 of them were timeouts caused by a defect in our own measurement, fixed the same morning. About 244 were infrastructure names with no website, and about 100 actually refused. Since then every delivery figure reads *refused from this vantage*, never *blocked*. The census figures now live in the census posts, starting with [which AI crawlers the top 10,000 sites block most](/blog/which-ai-crawlers-top-sites-block-most-october-2026), and on [the dataset page](/data).

## Rules, receipts and disputes in public #

The rulebook lists every finding CrawlCheck can raise; a finding that is not in it cannot appear in a report. A fix now ends in a signed receipt holding the before and after readings. When a receipt is lined up against a site's own traffic or leads, the page says plainly that a coincidence in time is not proof.

Anyone can dispute a finding. A site owner's dispute is accepted at once; anyone else first proves they control the domain. The finding is re-checked against its original evidence and a fresh reading, and the verdict (upheld, corrected, fixed since, vantage-dependent or inconclusive) is signed. The first resolution on the ledger was against one of our own sites, and it was upheld. Signed records and daily seals are listed on [the proof page](/proof).

## Second and third networks #

Independent observers on separate networks, one of them the browser extension, now re-read pages. A finding two networks reproduce reads *cross-vantage confirmed*; one they disagree on reads *observer disagreement* and is shown as such.

## Developers, privacy and security #

A typed SDK with the offline verifier shipped, along with published test fixtures so a third-party verifier can prove it agrees with ours. No median, percentile or count is shown for a group of sites too small to stay anonymous. An outside security audit of the running service produced eight items, all closed.

## Pricing and the site #

Two paid lines, Evidence and Intelligence, are described on [the pricing page](/pricing); every surface they gate is new. Findings and guides now read as one library, the [glossary](/glossary) gives each term its own page, and the navigation lost half its items without losing any page. Scoring did not change. Two rules were revised and say so on every card they touch: robots.txt matching follows RFC 9309 wildcards and anchors, and a stale-cache finding is confirmed before it is reported.

**Live, as you read this:** the corpus now holds 582,334 domains across 9,306 scans. The figures in this piece were measured on the date above; this line is not.

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Frelease-notes-3-october-2026-resolve-protocol-census-rulebook%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Frelease-notes-3-october-2026-resolve-protocol-census-rulebook%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Release%20notes%203%20Oct%202026%3A%20Resolve%2C%20census%2C%20rulebook)[Email](mailto:?subject=Release%20notes%203%20Oct%202026%3A%20Resolve%2C%20census%2C%20rulebook&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Frelease-notes-3-october-2026-resolve-protocol-census-rulebook)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### What is CrawlCheck Resolve?

One signed answer about a domain that an AI agent can read before fetching, quoting or acting on it. It covers declared crawl policy, what each crawler identity was served, machine files, identity confirmation and open findings, with no overall score.

### Can a Resolve answer be verified without contacting CrawlCheck?

Yes. Every answer is signed with a published key, and the offline verifier checks the signature with no call to crawlcheck.io.

### What was corrected in the first census write-up?

It said the homepages that did not answer were walling off datacenter traffic. Many were timeouts caused by a defect in CrawlCheck's own measurement, fixed the same morning; others were infrastructure names with no website. Delivery is now reported as refused from this vantage, never as blocked.

### Who can dispute a CrawlCheck finding?

Anyone. A site owner's dispute is accepted at once; anyone else first proves control of the domain. The finding is re-checked against its original evidence and a fresh reading, and the verdict is signed.

### Did scoring change in this release?

No. Two rules were revised, robots.txt matching and the stale-cache check, and each card they touch says so.

## Related findings

[2026-10-14 Release notes 14 Oct 2026: doors, weekly figures, trial 14 Oct 2026: crawlcheck.io reorganised into four audience doors, signed weekly editions for every live figure, a 30-day Watch tria](/blog/release-notes-14-october-2026-doors-signed-weekly-figures-watch-trial)[2026-09-29 Release notes 29 Sep 2026: finding states, sitemap, trust 29 September 2026 on crawlcheck.io: findings track their history across scans, the sitemap is split in five, and trust, terms and](/blog/release-notes-29-september-2026-finding-states-sitemap-index-trust-center)[2026-10-15 Agent conduct, October 2026: 34,676 verified requests Agent conduct on crawlcheck.io, 12 Aug to 15 Oct 2026: 34,676 verified requests, 9 of 12 crawlers read robots.txt, 3 never did, 84](/blog/agent-conduct-october-2026)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [/](/) · [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) · [Staple](/spec/staple) · [Conduct: Bingbot](/agents/conduct/bingbot) | **Registry:** [peerpush.com](/registry/capabilities/peerpush.com) | **Datasets:** [Technologies: Hotjar](/datasets/technologies/hotjar) · [Technologies: Font awesome](/datasets/technologies/font-awesome) | **Blog:** [Agent conduct, October 2026: 34,676 verified requests](/blog/agent-conduct-october-2026) | **Glossary:** [render dependence](/glossary/render-dependence) · [Vector database](/glossary/vector-database)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /blog/release-notes-3-october-2026-resolve-protocol-census-rulebook.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.