How CrawlCheck works
In one sentence
How it works is the page that puts every CrawlCheck step in order: a site is measured from outside as the crawlers see it, its declarations are read as written, published rules decide the findings, independent observers corroborate the readings, a signed answer resolves one decision per action (read, cite, connect, transact), and anyone can verify the whole chain offline with one script and the published keys. Site owners read the same measurement as a free A–F audit.
Scan a domain — free Ask before your agent acts → Paying never changes a reading. A licence adds the evidence behind each finding and the record over time — never a different answer.
From a domain to a signed answer
Read top to bottom. Nothing is guessed at any step: a section that could not be measured says so, and a decision built on it says why.
- 1
1 · Measure
Observe what the site actually serves
15 crawler identities fetch the homepage and every reply is kept: status, bytes, words, redirects. Five machine files are read on both hosts. Every body is hashed into a manifest that is sealed to Bitcoin daily.
- 2
2 · Measure
Read what the site declares
robots.txt is decided per RFC 9309 for 114 named agents; Content-Signal and rights lines are read as written; capability declarations are collected: MCP server cards, agent-skills indexes, API catalogs, A2A cards. Declared is kept apart from observed, always.
- 3
3 · Decide
Decide the findings by published rules
77 rules, each with its inputs, severity, test and fix on a public page. Every finding carries the rule revision that decided it and the fetches it read, so it replays over the sealed bytes. No licence, plan or payment moves one.
- 4
4 · Decide
Corroborate with independent observers
Enrolled observers on other networks re-read the same URL and sign what they got. Readings are compared field by field; a refusal that only one network sees is named as such. Anyone can enrol an observer with one key, and anyone can file a dispute.
- 5
5 · Answer
Resolve: one answer, one decision per action
Every section of the answer is declared, observed or not measured, with when. From it, a decision for read, cite, connect and transact: allow, warn, needs a human, or block. An agent’s own policy can make a decision stricter, never looser.
- 6
6 · Sign
Sign it, so anyone can verify it offline
Answers, receipts, exports and resolutions are signed with Ed25519 under published keys. One script verifies any of them with no account and no call to us. A site can check an agent’s receipt by its digest before serving the request.
The measurement, scored on a live report.
Rings are the stage scores a site owner reads — reach, read, quote — and bars the sections inside each. The same measurement feeds the signed answer. Open the report →
Five decisions, and what each one means
One per action — read, cite, connect, transact — decided from the answer’s own sections at signing time, and tightened by the agent’s own policy.
| Decision | When it is given | What the agent does |
|---|---|---|
| allow | The measured sections meet the action’s minimum assurance and nothing contradicts them. | Proceed. Keep the receipt with the action. |
| warn | Reachable, but a stated doubt remains: an agent-specific robots rule, an unmeasured section, a stale reading. | Proceed only if your own policy accepts the named doubt. |
| needs a human | The action is transact, or your policy requires approval for it. | A person approves before the agent acts. |
| block | Refused, disallowed for your agent, suspected impersonation, or on your deny list. | Do not act. The answer names an alternative when one was observed. |
| unsupported | The site declares nothing to connect to. | Nothing to do here; read or cite instead if those allow. |
What holds it together
Four properties turn a measurement into something a stranger can rely on.
Rules
Nothing is decided in private
Every finding code, its test, severity and fix are published, and the list is computed from the running scanner rather than typed.
Evidence
Every byte is kept and sealed
Each fetched body is hashed into a manifest; each day’s manifests are anchored to Bitcoin through OpenTimestamps. A reading can be replayed years later.
Observers
More than one vantage
Independent networks read the same URL and sign their readings. Agreement and disagreement are both published, per field.
Verification
Trust the maths, not us
Signed with published keys, verifiable offline by one script, with a public conformance suite that any verifier can be run against.
Public, and what a licence adds
Everything an agent needs is public and free: the answer, the decision, the rules, the keys, the census, the SDKs. A licence buys the depth a site owner needs: the evidence behind every finding, the record over time, and receipts for fixes.
PUBLIC
- Scan any domain: grade, every finding named, the top one in full
- Resolve one signed answer per domain, and a decision per action
- Rulebook, keys, conformance suite, census and the observer protocol
- SDKs on npm and PyPI with the offline verifier bundled
LICENCE
- Evidence behind every finding: the bytes, the locator, the rule replay
- The record over time: fingerprints, change receipts, the daily seal per domain
- Fix receipts a client can verify, and outcomes after a fix
- API quotas, batch scans, webhooks and a CI gate