How CrawlCheck is run, in lines an agent can check.
In one sentence
The CrawlCheck trust center states how the service is run so each line can be checked from outside: every Resolve answer and decision receipt is signed with Ed25519 and logged in a witnessed append-only log; data is stored on Cloudflare’s network with a sealed off-account copy on GitHub; a shareable report lives 90 days in the fast store and then in the sealed archive, a decision receipt until a day after it expires, and a domain’s measurement row until it opts out; nine named outside services each receive only what is listed; the site enforces HSTS, a nonce-based Content Security Policy and frame denial; and every day of reports is anchored to Bitcoin.
An agent that acts on a CrawlCheck answer is trusting how the service is run. This page says where data lives and for how long, what reaches anyone else, how the site is secured, and which record proves each answer — and every line is something the running system enforces and an outsider can test.
1
network: Cloudflare, plus a sealed copy on GitHub
9
outside services, each named below
Ed25519
signature on every answer and receipt
1
root a day anchored to Bitcoin
What an agent can verify, and where
Each check runs on the reader’s own machine from published keys and files.
| Record | What it proves | Check it at |
|---|---|---|
| A Resolve answer | Signed by CrawlCheck and unchanged since | /keys, the offline verifier |
| A decision receipt | This decision was issued for this answer, valid until its expiry | /api/v1/decisions/<sha256> |
| The answer log | An answer was entered in the append-only log and not rewritten | /api/v1/log and its outside witness |
| A report | It existed in this form by that day | /proof and the day’s .ots file |
| Our crawler requests | A request claiming to be CrawlCheck really is | the Web Bot Auth key directory |
Where data lives
Everything runs on Cloudflare’s network: the Workers that serve the site and measure, KV for records, and R2 object storage for evidence and archived answers. There is no other server and no other database. One copy leaves Cloudflare on purpose: each sealed day is mirrored nightly to a private GitHub repository, so the history survives the loss of the account. crawlcheck.io is owned by VSNARY.
How long it is kept
Every row expires on its own or on the domain’s opt-out; nothing is kept for want of a rule.
| What | Kept for | Why |
|---|---|---|
A shareable report at /r/<id> | 90 days in the fast store, then the sealed archive | So the link keeps working for whoever it was shared with, and receipts and disputes can point at it. The verified owner deletes it at /manage/data; a robots.txt opt-out hides it. |
| One measurement row per scanned domain | Until the domain opts out | So a re-scan can show what changed. Opt out with one robots.txt line. |
| The evidence behind a report: the stored record, the bytes each crawler received, the signed manifest and the ids that link them | Until the domain opts out | So every published conclusion can still be traced to what was received after its page expires. Read it at /api/verify?id=<id>&from=archive. |
| The answer an agent receives for a domain | 45 days ready to serve; every signed answer archived | Served in milliseconds; the archive lets a receipt be checked against the exact answer it was made from. |
| A decision receipt | Its validity (5 minutes for a transaction, 1 hour to connect, 24 hours to read or cite) plus one day | So the visited site can check a receipt an agent presents. |
| A first look at a never-measured domain | 6 hours | Answers within a second, then the full scan replaces it. |
| Counts of decisions, receipt checks and agent arrivals | 400 days | Counts only, for the adoption and share-of-preflight figures. |
| A quote request sent through a site’s lead form | 180 days | Delivered to the business it was addressed to, then expires. |
| A comment waiting for review | 30 days | Published or discarded by a person; unreviewed ones expire. |
| Search Console and Business Profile figures you connect | 400 days | Year-on-year comparison. Read-only; disconnecting stops collection. |
| A signed-in session (customer or owner) | 12 hours | Cookies are HttpOnly, Secure and SameSite=Strict. |
| Everything held about a domain you hold a licence for | Until you delete it | Export it, then delete it, from /manage (Your data). The deletion leaves a signed record of what was removed, counts only; the next scan starts a new history. |
| Recent requests to this site, and recent crawler requests, for the owner | The last 200 of each | A rolling tail with the requesting address, for abuse checks and crawler verification; never published and never exported. |
What is published per domain: the answer an agent is told (in the daily snapshot, CC BY 4.0), the directory’s vendors and the census frame. A domain that opts out leaves the next snapshot. A business found with a defect is not named in anything we write until its owner has been contacted. Everything a scan sends and keeps.
How the site is secured
Check any of these yourself: curl -sI https://crawlcheck.io/ prints every header.
| Control | Setting |
|---|---|
| Transport | HTTPS only. Strict-Transport-Security: max-age=31536000; includeSubDomains; preload |
| Content Security Policy | Nonce-based script-src with strict-dynamic, object-src 'none', frame-ancestors 'none', forms may post only to this site and Stripe checkout. Violations are reported to /api/csp-report. |
| Framing and sniffing | X-Frame-Options: DENY, X-Content-Type-Options: nosniff |
| Referrer and permissions | Referrer-Policy: strict-origin-when-cross-origin; camera, microphone, geolocation, USB, serial, Bluetooth, payment and topics APIs all disabled by Permissions-Policy |
| Certificates | CAA records limit issuance to six named authorities and send incident reports to hello@crawlcheck.io |
| Fetch safety | Scan requests refuse private, loopback and link-local destinations, never sign in, and never submit a form |
| Keys | A licence key is a credential. Send it in the x-crawlcheck-key or Authorization header rather than a URL, where it would land in logs; an API key can be revoked through the API at any time. |
| Record | What it does |
|---|---|
| SPF | Only Google Workspace and Cloudflare Email may send as crawlcheck.io. |
| DKIM | Mail this service sends is signed for crawlcheck.io and checked by the receiving server. |
| DMARC | p=quarantine for the domain and every subdomain, applied to 100% of mail; aggregate reports come back to us. |
| MTA-STS and TLS-RPT | A published policy asks sending servers to use verified TLS, and TLS-RPT sends us a report of every failure. It runs in testing mode, which reports failures without refusing mail, until the reports show a clean record. |
The archive rebuilds the evidence, checked every week
Every week a GitHub Actions run (proven to be this repository’s workflow by GitHub’s own signed token, so no secret is stored for it) takes the 10 oldest archived reports and the 10 oldest that carry sealed evidence, and resolves each one twice: through the 90-day working copies, and from the permanent store alone. The record, the report and decision nodes, the manifest, every verifier check, and two findings’ full evidence paths must come out identical. Any difference fails the run.
Latest green run: 2026-10-05 18:31 UTC · 20 reports (the oldest, and the oldest with sealed evidence), 32 decisions and 19 evidence paths, all identical · 9–58 days old · report ids: 3asjeiykt6 hrvk12zsu4 hi4gqmq8rf 5qnug9eatl w5dlx3gl7n v9yw4sk4ds rraa9yzomy zkete9dxc3 tyne9k1ibo lzo9858sr3 kzmd3h5n05 ovdx0w6ewz d9w2ht6wkd 8lcwhp9cwj sur040a3j0 zp4r4takhm 42smg2hxgt 6sw1yjb65y w1vuzsbuz4 qi79w5cuvt · the full result
Every report is timestamped
Proves
A report existed in exactly this form on that day and has not been edited since. Each report is reduced to its measurements and hashed, and each day’s hashes are folded into one root anchored in Bitcoin with OpenTimestamps. A day whose scheduled seal did not complete is sealed on the next hourly run and marked late.
Does not prove
That the measurement was right. Time and integrity only.
See the chain and download any day’s proof →
Outside services
What each one receives, and nothing more.
| Service | Used for | What it receives |
|---|---|---|
| Cloudflare | Runs and stores everything: the Workers that serve the site and measure, KV, R2 object storage, Durable Objects, Analytics Engine, Vectorize, DNS, outbound email, the open-weights models in the benchmark and the headless browser | Everything the site stores or serves |
| GitHub | An off-account copy of each sealed day, the weekly rebuild test, the transparency-log witness, the independent observer runs and our deploy workflows | Each sealed day (domains, what each crawler received, digests) in a private repository; signed log heads; observers' readings of public pages. Nothing from an account |
| Stripe | Checkout and billing | What you enter on Stripe's own checkout page; we never see a card number |
| Google Workspace | Our mailbox, hello@crawlcheck.io | Mail you send us |
| Google APIs | PageSpeed and Chrome UX field data; Search Console and Business Profile when you connect them; Gemini for answer checks and the benchmark; YouTube data for the YouTube module; place lookups; public DNS as a fallback resolver | A public URL, domain, business name or test prompt; your Google data only after you grant access |
| Google Fonts | Two typefaces on every page | Your browser fetches the font files from Google, so Google sees that browser's address and user agent |
| OpenTimestamps calendars | Anchoring the daily seal to Bitcoin | One hash per day; no report content |
| OpenStreetMap Nominatim and Photon | Turning a service-area city into coordinates | A city name |
| Wikidata, the Internet Archive and RDAP | An entity's identifiers, a page's history, when a lookalike domain was registered | A public name, URL or domain |
Reporting a problem
Email hello@crawlcheck.io with the subject line Security report. We acknowledge within three business days. Scope, rules and safe harbour are on the security page; the machine-readable version is security.txt.
What we collect →Proof and timestamps →Subprocessors and data location →Signing keys →Security →Terms →Who runs this →