Specification · v1

Open observer protocol

How anyone can run an independent observer for CrawlCheck: enrol a key, fetch tasks, fetch the pages from your own network, sign what you received, and have it corroborated beside every other observer’s reading. No GitHub, no Cloudflare and no account are needed; the reference implementation is one Node file with no dependencies.

Reference implementation · Fixtures · Observer directory · Corroboration API

1. Enrol

Make an Ed25519 key. POST /api/observe/enroll with {"request": {...}, "sig": "..."} where request is {kind: "crawlcheck-observer-enrolment", v: 1, observer_id: "ind:<name>", operator, network, contact?, software?, key: {kty: "OKP", crv: "Ed25519", x}, requested_at} and sig is the base64url Ed25519 signature over "crawlcheck-observer-enrol-v1\n" followed by the canonical JSON of request (keys sorted, no whitespace). The id is bound to the first key that claims it. The Worker records the network the request came from (ASN and organisation from the connection itself) next to the network you declared, and publishes both.

2. Fetch tasks

GET /api/observe/tasks with headers x-cc-observer-id, x-cc-observer-time (ISO, within 5 minutes) and x-cc-observer-sig: your signature over "crawlcheck-observer-tasks-v1\n" + id + newline + time. Each task names a URL and the exact request profiles (headers per identity) to use; profile_sha256 pins them.

3. Observe and sign

Fetch the task URL once per identity with exactly those headers, following redirects. Build the envelope {v: 1, kind: "crawlcheck-observation", observer_id, software, run, vantage, clock, profile_sha256, task, fetches[], key} with key equal to your enrolled key, and sign "crawlcheck-observation-v1\n" + canonical JSON of the envelope. POST {envelope, sig, bodies} to /api/observe/signed; bodies maps each fetch’s sha256 to the base64 bytes (at most 2 MB each), which are stored by hash and re-hashed on receipt.

4. Verification

• The envelope signature verifies under the key the directory lists for that observer id, and the envelope carries exactly that key.
• observed_at is within 30 minutes of receipt; clock skew is recorded.
• The request profiles hash to the task’s profile_sha256, or the reading is marked as made with other profiles.
• Every body hashes to the sha256 its fetch names.
• The network each submission arrived from is recorded beside the declared vantage.

5. Corroboration

Every observer’s latest reading of a URL appears beside the primary scan, field by field (status, visible words, blocker, redirect host), and each field is marked agree or differs. Nothing is averaged and no reading is dropped. Revoked observers’ readings stay listed as revoked.

A live multi-vantage record

chroma.com, homepage, read by 3 observers on 3 networks (cloudflare, microsoft-azure, independent). Each cell is that observer’s value; a field agrees only when every observer that read it reports the same.

57 fields agree, 0 differ. JSON: /api/corroboration?domain=chroma.com (by_field).

6. Run an observer on your WordPress site

The WordPress observer (one PHP file, MIT, sha256 6968f817692e5856…) is this protocol in about 150 lines: on first run it makes an Ed25519 key with libsodium and keeps it in your site’s options table, serves its key id at /.well-known/crawlcheck-observer.txt, enrols with site=<your domain>, and once an hour reads up to six public pages of other domains from your host’s network and posts a signed reading of each. It sends no cookies, submits no forms, calls no tools, and nothing about your visitors leaves your site. Install it as a plugin or paste it into a code-snippets plugin; deactivate it to stop. Your site then appears on the observer mesh as a customer observer, and every reading it makes is counted under your host’s network in the answers for the domains it read.

Enrolled independent observers

Fixtures

/observer/fixtures.json holds signed envelopes — valid, tampered after signing, signed by a key the directory does not list, carrying a key other than the enrolled one, and an enrolment with a forged signature — each with the verdict a conforming verifier must return. node observer.mjs verify fixtures.json runs them; so does POST /api/observe/check, which applies the Worker’s own checks to a fixture without storing anything.