CrawlCheck
Assurance

Availability, receipts, observers, coverage and adoption, each with the denominator under it.

What an agent depends on when it asks before acting: the service answering in time, a receipt it can check without trusting us, observers it can tell apart from us, coverage counted honestly, and evidence of who actually uses it. Every number here is read from the running system when the page loads; the same document is /api/v1/assurance.

0400,000800,000attemptedattempted: 700,325700,325measuredmeasured: 496,554496,554full scansfull scans: 6,2686,268registry-listedregistry-listed: 1,1011,101
Denominator: 700,325 domains CrawlCheck has tried to measure · Scale: domains, linear, 0–800,000 · Window: as of 2026-11-08 13:35 UTC, cumulative

Definition

Attempted
A domain CrawlCheck tried to read, from the census, the registry, a lookup or a scan.
Measured
A domain with a stored, signed record of what was served.
Full scan
The on-demand deep read: every section, every finding with its evidence.
Registry-listed
A domain with a record in the public registry, claimed or not.

What this does not prove

  • Measured means read and recorded, not right: every finding carries its evidence so it can be checked.
  • Coverage counts what CrawlCheck was asked about or seeded with, not a sample of the web.
  • A full scan is a moment; the record says when it was taken.
  • The keys, observers and eligibility checks further down are listed as configured; the status page says whether they are answering now.

Check it yourself

The numbers on this page come from the signed JSON at /api/v1/instruments/assurance/now.json, rebuilt every few minutes; each copy carries its own sha256 and an Ed25519 signature by key 1hHmE3fo1rg5…. For a copy that never changes, cite a weekly edition. Verify either with no call to us:

node crawlcheck-verify.mjs --online assurance.json

The verifier: /crawlcheck-verify.mjs, zero dependencies.

Editions

One sealed edition per ISO week: the signed reading as first taken that week, never edited. Feed: Atom · all instruments: /api/v1/instruments.

How a decision receipt maps to EU AI Act Article 12 and NIST AI RMF, as a mapping and not legal advice: decision receipts as compliance evidence. A frozen public suite of 27 cases, run alone and with a signed answer: the agent safety benchmark.

degraded

service now (status)

2

signing keys, by purpose

1 of 15

observers not run by us

0

decisions to verified AI agents, 30 days

1 · Availability

Targets and what was measured

Last 30 days. /api/slo

ObjectiveTargetMeasuredSamplesp50 / p95Result
Resolve answers without a server error99.9%100%16,080206 / 625 msmet
met at 95% confidence
Resolve answers for a measured domain within 300 ms in the Worker95%87.4%16,080206 / 625 msmissed
missed at 95% confidence
Preflight decisions without a server error99.9%100%3,384220 / 668 msmet
met at 95% confidence
Preflight decisions within 300 ms in the Worker95%69.9%3,384220 / 668 msmissed
missed at 95% confidence
A first answer for a never-measured domain within 2 seconds in the Worker95%87.6%7841139 / 2761 msmissed
missed at 95% confidence

A missed objective stays published until the window rolls past it. Open incidents: 0; the signed incident log is at /api/status/log.

2 · Before acting

A valid signature is not permission

Six separate answers; act only when all hold.

CheckFails when
integrity_valida byte of the receipt changed after signing
signature_validthe Ed25519 signature does not verify over the digest
issuer_trustedthe key is not in CrawlCheck’s published directory: a perfect signature from anyone else’s key is refused
freshthe receipt is past expires_at (each action has its own validity; transact is the shortest)
scope_validthe receipt names a different domain or action from the one you are about to take
decision_permitsthe decision is block or unsupported; warn without your policy accepting it; require_confirmation without a human’s yes
curl -s -X POST https://crawlcheck.io/api/v1/preflight -H 'content-type: application/json' \
  -d '{"domain":"example.com","action":"read"}' -o receipt.json
curl -sO https://crawlcheck.io/crawlcheck-verify.mjs
node crawlcheck-verify.mjs receipt.json --act example.com:read      # exit 0 only when eligible_to_act

The SDK’s guard() makes the same checks in your code; the quickstart puts it inside the function that performs the action, so the model cannot route around it. Keys: requests and documents are signed with separate keys.

3 · Independence

Who runs each observer and witness

14 of 15 are run by CrawlCheck.

Different networks are not different operators. An observer we run from another cloud shows whether a result depends on where it was measured from; only an operator outside CrawlCheck can show it does not depend on us. Each row says which it is.

ObserverOperatorNetworkVerifiesCode · credentials · deployLast readingLimitation
crawlcheck-worker
active
CrawlCheck (VSNARY) (us)CrawlCheck primary networkevery primary measurementours · ours (Worker secret) · ours—this is CrawlCheck itself: it corroborates nothing
gha-azure-1
active
CrawlCheck (VSNARY) (us)GitHub-hosted runners on Microsoft Azure; not any crawler operator's network, so it corroborates a vantage question, never an operator's own crawlthe homepage under each identity, from a second networkours · a new key per run, bound to our workflow by a GitHub token · ours (repository workflow)2026-11-07 15:49a different network and a third party (GitHub) attesting which workflow ran; not a different operator
aws-us-east-2
active
CrawlCheck (VSNARY) (us)Amazon Web Services, US East (Ohio); not any crawler operator's network, so it answers whether a site treats this network or country differently, never what an the homepage under each identity, from a second networkours · a new key per run, bound to our workflow by a GitHub token · ours (repository workflow)2026-11-07 15:49a different network and a third party (GitHub) attesting which workflow ran; not a different operator
tor-exit
active
CrawlCheck (VSNARY) (us)the Tor network: an exit relay Tor chooses (address, network and country recorded per reading); it answers whether a site treats anonymising networks differentlwhat a crawler is served from a visitor's own connectionours (the extension) · one key per install, made on the device · installed by each person2026-11-07 15:48the vantage is the visitor's; the code is ours; an install key proves one install, not one person
ext:*
active
people running the CrawlCheck extension (version 1.0.4 and later) with sharing switched onthe visitor's own connection (home, office or mobile), recorded as country and network name only; the address is never storedwhat a crawler is served from a visitor's own connectionours (the extension) · one key per install, made on the device · installed by each person—the vantage is the visitor's; the code is ours; an install key proves one install, not one person
gha:*
active
any GitHub repository whose workflow runs the crawlcheck-core Action with observe: true (or posts the same envelope); id gha:<owner>/<repo> (us)the runner the workflow ran on: GitHub-hosted (Microsoft Azure) or the repository's own self-hosted runner, measured from the connection each reading arrives onwhat a crawler is served from a visitor's own connectionours (the extension) · one key per install, made on the device · installed by each person—the vantage is the visitor's; the code is ours; an install key proves one install, not one person
ind:cc-cosig1-test
revoked
CrawlCheck (VSNARY) cosig1 end-to-end test; retired after the test (us)test container, one-offreadings of queued URLs under the open observer protocolthe open protocol client · an Ed25519 key made on the observer's machine · whoever runs it—run by CrawlCheck from another network: network diversity, not an independent operator
ind:cli-2ns-x7r1zy
unknown
CrawlCheck (VSNARY), PHP client test from a Google Cloud container (us)this machine's networkreadings of queued URLs under the open observer protocolthe open protocol client · an Ed25519 key made on the observer's machine · whoever runs it—run by CrawlCheck from another network: network diversity, not an independent operator
ind:gcp-session-1
unknown
CrawlCheck (VSNARY), run from a Claude cloud session (us)Google Cloud egress (AS396982), not Cloudflare, not Azurereadings of queued URLs under the open observer protocolthe open protocol client · an Ed25519 key made on the observer's machine · whoever runs it—run by CrawlCheck from another network: network diversity, not an independent operator
ind:gcp-tv-1
unknown
CrawlCheck (VSNARY), test vantage run from a Google Cloud container (us)Google Cloud (AS396982), Columbus USreadings of queued URLs under the open observer protocolthe open protocol client · an Ed25519 key made on the observer's machine · whoever runs it—run by CrawlCheck from another network: network diversity, not an independent operator
ind:google-llc-iowa-e94e42
quarantined
CrawlCheck (VSNARY) - installer test from a Google Cloud container (us)AS396982 Google LLC, Council Bluffs, Iowa, USreadings of queued URLs under the open observer protocolthe open protocol client · an Ed25519 key made on the observer's machine · whoever runs it—run by CrawlCheck from another network: network diversity, not an independent operator
ind:installtest-gcp
quarantined
CrawlCheck (VSNARY) installer test (us)AS396982 Google LLC, Columbus USreadings of queued URLs under the open observer protocolthe open protocol client · an Ed25519 key made on the observer's machine · whoever runs it2026-10-08 20:40run by CrawlCheck from another network: network diversity, not an independent operator
ind:refcheck-gcp-
quarantined
CrawlCheck (VSNARY) reference check (us)Google Cloud containerreadings of queued URLs under the open observer protocolthe open protocol client · an Ed25519 key made on the observer's machine · whoever runs it2026-10-08 21:13run by CrawlCheck from another network: network diversity, not an independent operator
ind:vsnary-wp-1
active
VSNARY (operator of crawlcheck.io) (us)managed WordPress hosting: the WordPress server of tree-d.examplereadings of queued URLs under the open observer protocolthe open protocol client · an Ed25519 key made on the observer's machine · whoever runs it2026-11-08 13:27run by CrawlCheck from another network: network diversity, not an independent operator
log-witness
active
GitHub (identity token) running CrawlCheck's workflow (us)GitHub-hosted runnerseach signed head of the Resolve log, co-signed with a GitHub-issued token naming the workflow that saw itours · GitHub's OIDC token for that run · ours (repository workflow)—GitHub attests which workflow ran and when; it does not recompute the log. A split view is caught by clients comparing heads (gossip)

Anyone can run an observer under the open observer protocol; an observer run by someone else is the one thing that turns network diversity into independence.

4 · Coverage

What each number counts

DomainsCount
attempted by the census700,325
with at least one reading (“sites measured”)496,554
full scans, graded6,268
in the public registry (full scan, publishable, not opted out)1,101

attempted = every domain the census tried; measured = domains with at least one reading (most are the census's lighter reading); full scans = graded reports; registry = full scans that can be published and have not opted out. Resolve answers for any domain; a never-measured one gets a first, partial answer marked as such.

MCP serversCount
records in the official MCP Registry, imported300
with a remote endpoint262
probed this cycle (handshake only, never tools/call)45,656
completed the handshake and listed tools22,936
completed the handshake with no tools1,239
asked for credentials15,236
errored, unreachable or not MCP6,168
tool lists scanned for poisoning866
flagged by the scan0
tool lists changed since first seen3,749

indexed is not connected, and connected is not safe: a clean tool scan means no known poisoning pattern was found, not that none exists. Cycle started 2026-11-01.

5 · Adoption

Who actually asks

Our own traffic is counted apart, never as adoption.

Last 30 daysCount
signed decisions to verified AI agents (operator ranges)0
signed decisions to other callers1,523
signed decisions to us (tests, probes, our sites)844
receipt checks by sites we do not run221
receipt checks by our own sites4

Calls to the agent surfaces (MCP, Resolve, preflight, Visitor Resolve, outcome) 2026-11-01 to 2026-11-07: 18,701 from outside, 3,821 ours. Outside, by caller: other 6,337 · mcp-client 4,675 · node 1,939 · browser 1,374 · bot:other 1,315 · python 630 · http-client 474 · bot:BrickBlueBot 316 · no-ua 315 · openai 240 · sdk-python 232 · claude 202 · bot:MJ12bot 175 · bot:Amazonbot 152 · bot:SealKeeperBot 81 · bot:WellknownBot 40 · bot:InvokeRankBot 37 · bot:Maps-bot 28 · bot:agent-tools.cloud-crawle 28 · browser-app 22 · bot:PluginboardBot 19 · bot:GolemreachTrustBot 16 · bot:YandexBot 14 · bot:ElectraIndexBot 9 · bot:Faceabot 8 · bot:CapabilityIndexBot 4 · bot:KakaoBot 3 · bot:Bytespider 2 · bot:brickroad-bot 2 · bot:SemrushBot 2 · bot:Baiduspider 2 · bot:AhrefsBot 2 · bot:ExtOpsBot 1 · bot:RiverBot 1 · perplexity 1 · bot:APIEvangelistBot 1 · bot:DotBot 1 · bot:DataForSeoBot 1.

ours = our key, owner session, own sites, probes or self-tagged tests; other = everyone else and may still include an untagged test of ours; verified_agent = a caller naming an AI agent from inside that operator's published ranges, which we cannot pad. Caller classes are read from the user agent and recorded from 2026-10-10. The day-by-day series is /api/v1/adoption.

Funnel, last 30 days

StageCountOfShare
decisions asked for by callers that are not us
verified agents plus other callers; other may still include an untagged test of ours
1,533——
of those, from an AI agent verified by its operator's ranges01,5330%
distinct agent names asking (self-declared)
counted from 10 Oct 2026; an agent that names itself the same way is one
15——
receipts checked by a site we do not run
a presented receipt is only visible when the site checks it, so presented and checked are one stage
2211,53314.4%
distinct sites checking receipts2——
agent names asking on 2 or more days51533.3%
sites checking on 2 or more days22100%

Our own traffic (our key, owner session, own sites, self-tagged tests) is never counted. Receipt checks recorded before 10 Oct 2026 were classed by host alone, so they are listed apart as host_only_receipts and excluded from every stage and from the goal.

6 · Operations

Witness, seals, drills and succession

Read from the running system; each row links to its source.

WhatLastMeasuredSource
Outside witness co-signs the log7 h agobatch 1,955 of 1,966 (11 behind) · hourly at :51 UTC on GitHub Actions, which can run latewitness
Daily seal after the day closes2026-11-077 of 7 days sealed · median 77 min after close, worst 78 · 7 in a Bitcoin blockproof
Revocation drill20 h agopassed in 1,008 ms · 14 of 14 recent runs passedself-test
Restore test (rebuild from archive)6 days agogreen · 20 of 20 identical, 0 mismatched · weekly, Mondays 03:17 MDT (09:17 UTC), and on demand; GitHub Actions, proven by GitHub's OIDC tokentest

daily: the incident switch on selftest.crawlcheck.io is flipped, Resolve must report the incident, a transact preflight must return block, and the domain must appear on the signed revocation list, then it is cleared. the oldest archived records rebuilt from the archive alone (the 90-day projections ignored) and compared byte for byte with what was served. delay = when the seal was written after the day closed; a seal reaches Bitcoin when a calendar's attestation is mined, usually within hours.

Succession

If the operator stops, this continues:

  • every signed answer, receipt and evidence bundle verifies offline with crawlcheck-verify.mjs and the published keys, with no call to CrawlCheck
  • each sealed day stays provable through its OpenTimestamps proof and Bitcoin block, which no one at CrawlCheck can alter
  • the daily signed snapshot and the census are public (CC BY 4.0) and can be mirrored by anyone
  • the transparency log's witnessed heads are held by every client that kept them

This stops:

  • new measurements and new signed answers
  • revocations and incident switches
  • the observer network's scheduling

no successor operator has been named yet; until one is, the guarantees above are the ones that survive the operator.

Generated 2026-11-08 13:35 UTC.