Specification · v1
Open observer protocol
How anyone can run an independent observer for CrawlCheck: enrol a key, fetch tasks, fetch the pages from your own network, sign what you received, and have it corroborated beside every other observer’s reading. No particular host, CI service or account is needed; the reference implementation is one Node file with no dependencies.
Reference implementation · Fixtures · Observer directory · Corroboration API
From your own CI (no enrolment)
A GitHub repository can be an observer with no key to keep: add observe: "true" to the crawlcheck-core Action and give the job permissions: id-token: write. Each run makes a key, GitHub signs a token bound to it naming the repository and workflow, and the id is gha:<owner>/<repo>. A self-hosted runner reads from your own network. The first verified run registers the repository by GitHub’s numeric ids. Readings are stored and listed at once; the vote counts after 7 days and 20 accepted readings, like every new observer.
1. Enrol
Make an Ed25519 key. POST /api/observe/enroll with {"request": {...}, "sig": "..."} where request is {kind: "crawlcheck-observer-enrolment", v: 1, observer_id: "ind:<name>", operator, network, contact?, software?, key: {kty: "OKP", crv: "Ed25519", x}, requested_at} and sig is the base64url Ed25519 signature over "crawlcheck-observer-enrol-v1\n" followed by the canonical JSON of request (keys sorted, no whitespace). The id is bound to the first key that claims it. CrawlCheck records the network the request came from (ASN and organisation from the connection itself) next to the network you declared, and publishes both.
2. Fetch tasks
GET /api/observe/tasks with headers x-cc-observer-id, x-cc-observer-time (ISO, within 5 minutes) and x-cc-observer-sig: your signature over "crawlcheck-observer-tasks-v1\n" + id + newline + time. Each task names a URL and the exact request profiles (headers per identity) to use; profile_sha256 pins them.
3. Observe and sign
Fetch the task URL once per identity with exactly those headers, following redirects. Build the envelope {v: 1, kind: "crawlcheck-observation", observer_id, software, run, vantage, clock, profile_sha256, task, fetches[], key} with key equal to your enrolled key, and sign "crawlcheck-observation-v1\n" + canonical JSON of the envelope. POST {envelope, sig, bodies} to /api/observe/signed; bodies maps each fetch’s sha256 to the base64 bytes (at most 2 MB each), which are stored by hash and re-hashed on receipt.
4. Verification
- The envelope signature verifies under the key the directory lists for that observer id, and the envelope carries exactly that key.
- observed_at is within 30 minutes of receipt; clock skew is recorded.
- The request profiles hash to the task’s profile_sha256, or the reading is marked as made with other profiles.
- Every body hashes to the sha256 its fetch names.
- The network each submission arrived from is recorded beside the declared vantage.
5. Corroboration
Every observer’s latest reading of a URL appears beside the primary scan, field by field (status, visible words, blocker, redirect host), and each field is marked agree or differs (visible words agree within 15%; every other field must be identical). Nothing is averaged and no reading is dropped. Revoked observers’ readings stay listed as revoked.
A live multi-vantage record
thekitchn.com, homepage, read by 3 observers on 3 networks (crawlcheck, microsoft-azure, vultr). Each cell is that observer’s value; a field agrees only when every observer that read it reports the same value, except visible words, which agree within 15% of the largest reading (a live page gains or loses a few words between two reads).
| Identity | Field | crawlcheck-primary crawlcheck | gha:emmanuelorta/crawlcheck-core microsoft-azure | ind:vsnary-wp-1 vultr | Agree |
|---|---|---|---|---|---|
| Unnamed client | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| GPTBot | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| ClaudeBot | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| OAI-SearchBot | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Claude-SearchBot | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| PerplexityBot | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| ChatGPT-User | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Claude-User | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Perplexity-User | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Googlebot | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Googlebot Smartphone | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Mobile browser | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Mobile browser (control) | status | — | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | — | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Bingbot | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Applebot | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Amazonbot | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Bytespider | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| Meta-ExternalAgent | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ | |
| CCBot | status | 403 | 403 | 403 | ✓ |
| words | — | — | — | — | |
| blocker | a CAPTCHA wall | a CAPTCHA wall | a CAPTCHA wall | ✓ |
39 fields agree, 18 differ. JSON: /api/corroboration?domain=thekitchn.com (by_field).
5b. Run an observer on any server, in one step
On a Linux server (Debian, Ubuntu, RHEL or Alpine; x64 or arm64), as root, or pasted as the server’s cloud-init / user-data when you create it:
curl -fsSL https://crawlcheck.io/observer/install.sh | sh -s -- --operator "Your name or company"The installer fetches Node 20 from nodejs.org and checks it against the published checksums, fetches the reference observer and checks it against its SHA-256, makes the key on the server (it never leaves), enrols, and adds an hourly cron job under its own user. Each server in a new network or country is a new vantage: a reading that agrees from a network CrawlCheck does not run on is what turns a single observation into a confirmed one.
5c. Private observers (licence holders)
Run the same observer inside your own networks with --private and your licence key in CC_LICENCE_KEY. Its id is prv:<name>; it enrols at POST /api/app/observers/enroll (the key is used once, to enrol, and is not kept on the server), fetches tasks at GET /api/observe/private/tasks and submits at POST /api/observe/private/signed, signed exactly as above. It is given only your licence’s domains, their subdomains and the URLs you queue. Its readings are kept under your licence, shown to you at GET /api/app/observers/readings?domain= and on /manage, and never listed here or counted in public corroboration, agreement checks or Resolve answers. Page bodies are read to compute each reading and are not stored.
5d. Co-sign MCP tool changes
The short version for operators: run an observer.
When CrawlCheck records that a public MCP server’s tools changed, every reading behind that record so far comes from one operator. An enrolled observer can confirm it from its own network. Ask for tasks with ?types=page,mcp_tools (the reference implementation does from version 1.2.0); a task of type mcp_tools names a server whose tools just changed, never the digest CrawlCheck recorded. Read it the way the sweep does: initialize, notifications/initialized, tools/list (first page), then end the session; no tool is called. Hash the list as the mcp-lockfile does: each tool’s sha256 over canonical JSON of {name, description, inputSchema, annotations}, then the sha256 of those hashes sorted and joined with newlines.
Build {v: 1, kind: "crawlcheck-mcp-tools-observation", observer_id, software, vantage, observed_at, task: {task_id, event_id, url}, result: {outcome, tools, tools_sha256, tool_hashes[], server, protocol_version}, key}, sign "crawlcheck-mcp-cosign-v1\n" + canonical JSON of it with your enrolled key, and POST {envelope, sig} to /api/observe/signed. The answer says whether your digest matched. Each change event then carries co_signatures (observer_id, digest, agrees, signed_at, network) and attested_by, the number of networks whose observers read the same digest; two observers on one network count once. The feed and the study say confirmed by N independent observers once N is above zero. A different digest is recorded too: the server may have changed again, or answered differently to your network. Every co-signature and its proof are at /api/observe/cosig?event=<id>. Tasks stay open for 48 hours after a change; at most 200 co-signatures per observer a day.
A co-signature is the observer’s signed statement of what its own read returned. It is worth what the observer’s independence is worth, which is why every one is listed with its observer and the network it arrived from.
Retire an observer
node observer.mjs retire signs {kind: "crawlcheck-observer-retirement", v: 1, observer_id, requested_at, reason} with the enrolled key and POSTs it to /api/observe/retire. The id stops receiving tasks and its co-signatures stop counting; its readings stay listed.
6. Run an observer on your WordPress site
The WordPress observer (one PHP file, MIT, sha256 6968f817692e5856…) is this protocol in about 150 lines: on first run it makes an Ed25519 key with libsodium and keeps it in your site’s options table, serves its key id at /.well-known/crawlcheck-observer.txt, enrols with site=<your domain>, and once an hour reads up to six public pages of other domains from your host’s network and posts a signed reading of each. It sends no cookies, submits no forms, calls no tools, and nothing about your visitors leaves your site. Install it as a plugin or paste it into a code-snippets plugin; deactivate it to stop. Your site then appears on the observer mesh as a customer observer, and every reading it makes is counted under your host’s network in the answers for the domains it read.
Enrolled independent observers
| Observer | Operator | Network (measured at enrolment) | Key id | Status |
|---|---|---|---|---|
| ind:cc-cosig1-test | CrawlCheck (VSNARY) cosig1 end-to-end test; retired after the test | Google LLC · AS396982 · US (google-cloud) | nZ2g-DrDwh5z0UAR… | revoked · unvetted |
| ind:cli-2ns-x7r1zy | CrawlCheck (VSNARY), PHP client test from a Google Cloud container | Anthropic, PBC · AS396982 · US (independent) | 2nS-X7r1ZyknyV7m… | active · unvetted |
| ind:gcp-session-1 | CrawlCheck (VSNARY), run from a Claude cloud session | Google LLC · AS396982 · US (independent) | rCUICgLBrBe1ZbzN… | active · unvetted |
| ind:gcp-tv-1 | CrawlCheck (VSNARY), test vantage run from a Google Cloud container | Google LLC · AS396982 · US (independent) | FflVrsDjtFnli52w… | active · vetted |
| ind:google-llc-iowa-e94e42 | CrawlCheck (VSNARY) - installer test from a Google Cloud container | Google LLC · AS396982 · US (independent) | -lbn7znC13WLhBRM… | active · unvetted |
| ind:installtest-gcp | CrawlCheck (VSNARY) installer test | Google LLC · AS396982 · US (independent) | sciAyOIAJDPAW-_W… | active · unvetted |
| ind:refcheck-gcp- | CrawlCheck (VSNARY) reference check | Google LLC · AS396982 · US (independent) | 5HYFbfKIlcftrn4R… | active · unvetted |
| ind:vsnary-wp-1 | VSNARY (operator of crawlcheck.io) | Vultr Holdings, LLC · AS20473 · US (independent) | BNcYAoHlDI7nPPGS… | active · vetted |
Fixtures
/observer/fixtures.json holds signed envelopes — valid, tampered after signing, signed by a key the directory does not list, carrying a key other than the enrolled one, and an enrolment with a forged signature — each with the verdict a conforming verifier must return. node observer.mjs verify fixtures.json runs them; so does POST /api/observe/check, which applies the service’s own checks to a fixture without storing anything.