What an AI agent should know about a domain before it reads, cites, connects or transacts, as one signed document. Any tool may read it, cache it, verify it offline and build on it. This page is the specification; the change log at the end is its history.
v1.2.0
this specification
1
document version; never changed in place
16
sections an agent can decide on
0
overall scores, on purpose
Three calls cover the path from a domain name to an action you can account for.
1 · Ask
GET /api/v1/resolve?domain= returns the signed answer; a never-measured domain comes back partial in about a second and upgrades when the full scan finishes. A site may staple its own answer in a header, and the DNS form needs no HTTP client.
2 · Decide
POST /api/v1/preflight turns the answer into one decision for the action you are about to take, signed, with the rules that decided it. The visited site can check the receipt you present.
3 · Verify
Recompute the digest, check the Ed25519 signature against the published key, and confirm the answer’s leaf in the witnessed log. No call to CrawlCheck is needed.
JSON Schema · machine-readable spec · API reference with worked examples · offline verifier · signing keys · Resolve over DNS · observer protocol
Every endpoint is in the OpenAPI document; these are the ones an agent acts on.
| Method | Path | Purpose |
|---|---|---|
| GET | /api/v1/resolve?domain= | One signed ResolveV1 answer. Also /v1/resolve/{domain} and /api/v1/resolve/{domain}. Cached at the edge for 5 minutes. Add as_of= for the answer that was in force at a past time, as it was signed then. |
| POST | /api/v1/resolve | Batch: {"domains": [..]}, 1-100 names, one ResolveV1 per name in the order sent. |
| GET | /api/v1/resolve-prefix/{prefix} | Private lookup: send 2-8 hex characters of sha256(domain) and pick your answer out of every measured domain under that prefix, so the domain you are about to act on is never sent. |
| POST | /api/v1/preflight | A signed decision for one action {domain, action, agent?, template?, policy?}: allow, warn, require_confirmation, block or unsupported, with reason codes and an expiry. |
| GET | /api/v1/decisions/{sha256} | The visited site checks a receipt an agent presents (the CrawlCheck-Receipt header names its digest). |
| GET | /api/v1/resolve/archive/{sha256} | The exact signed answer a decision receipt was made from. |
| GET | /api/v1/staple?domain= | A site’s own signed verdict to serve in the CrawlCheck-Staple header, so an agent needs no lookup. |
| GET | /dns-query | Resolve over DNS-over-HTTPS: a signed TXT record at <domain>._v.crawlcheck.io. |
| GET | /api/v1/changes?since= | Change feed: one event per section that moved when an answer was rebuilt. Poll with the last next_since; re-resolve only the domains listed. |
| GET | /api/v1/log | The append-only answer log: the latest signed tree head, inclusion proofs and the outside witness. |
| GET | /api/v1/snapshot | Every public answer, once a day, signed: for agents that want the whole set offline. |
| GET / POST | /api/v1/visitor | Visitor Resolve: for a site, whether a request really comes from the agent its user agent names. |
| POST | /api/v1/outcome | An agent reports what happened when it acted. Self-reported; counted, never used to change an answer. |
| GET | /api/v1/outcomes?domain= | What agents reported for a domain over 30 days, including disagreements with the answer. |
| GET | /api/v1/domain?domain= | The domain’s public registry record (RDAP) and its timeline. |
| GET | /api/v1/accuracy?domain= | What Google’s AI states about a claimed business against what the business publishes. |
MCP: the tool resolve_domain on https://crawlcheck.io/mcp returns the same document.
kind: "crawlcheck-resolve", v: 1. status is measured, partial (a domain never measured before, read live in about a second; assurance says how much was read and upgrade when the full scan replaces it), not_measured (a measurement is requested) or opted_out. Each section carries state: declared (what the site says about itself), observed (what CrawlCheck was served) or not_measured, and observed_at. There is deliberately no overall score: an agent decides on the section its action depends on, or takes the decision in actions.
| Section | Answers |
|---|---|
| crawl_policy | robots.txt as declared, per crawler, with signal conflicts |
| delivery | what each identity (browser, each AI crawler) was actually served: status, words, blocker, redirect |
| machine_files | llms.txt, sitemaps, well-known files: status and hashes |
| capabilities | what the site declares agents can do, by safety class, with verification and mismatches |
| mcp_registry | remote MCP servers listed for the domain and what each answered to the handshake |
| entity | who the site says it is and whether external sources confirm it back |
| self_claims | what the site says about itself (phone, address, hours, prices) and where each claim is published |
| impersonation | whether the domain looks built to be mistaken for a known brand |
| read_equivalence | whether the main text AI crawlers are served matches what a browser gets |
| content_clock | when the main content last changed, from a hash of the content alone |
| rights | one permission answer per use: train, ground, summarize, cite, transact |
| findings | open findings, each with its rule code and revision; the top one in full without a licence |
| actions | a decision for read, cite, connect, transact and administer, made from the sections above |
| absences | a signed record of what was looked for and not found, and where |
| freshness, observers, confidence, contradictions | how old the reading is, from how many vantages, and where they disagree |
| evidence, transparency | the report and manifest the answer was built from, and its leaf and inclusion proof in the answer log |
sha256 is the SHA-256 of the canonical JSON of every field except sha256 and signature (keys sorted at every level, no whitespace). signature.sig is Ed25519 over the message prefix crawlcheck-data-v1\n followed by that digest. The key is named by signature.kid and published in /.well-known/http-message-signatures-directory; retired keys stay listed so old answers still verify. The same key signs decision receipts, staples, DNS records and the log’s tree heads.
node crawlcheck-verify.mjs answer.json (Node 20+, no dependencies, no call to crawlcheck.io), or the same checks from @crawlcheck/sdk and crawlcheck on PyPI. It recomputes the digest, checks the signature against the published key and reports which check failed. transparency carries the answer’s leaf and inclusion path, which fold to a signed tree head an outside witness co-signs. Every answer is also a leaf in that day’s Merkle root, anchored with OpenTimestamps; /api/seal/lookup?id=&day= says whether a leaf is in the sealed root.
v in the document changes only when a field is removed or its meaning changes; such a change ships as a new version beside the old one, never in place. Added fields and added endpoints raise the minor number of this specification. Readers must ignore fields they do not know.
| Version | Date | Change |
|---|---|---|
| 1.2.0 | 2026-10-07 | Added to the document: status partial (a never-measured domain read live in about a second, with assurance and upgrade), and the sections impersonation, mcp_registry, content_clock, self_claims, read_equivalence, rights, actions (a decision per read, cite, connect, transact, administer), absences (signed absence) and transparency (the answer's leaf and inclusion proof in the append-only log). Added endpoints: preflight decisions POST /api/v1/preflight; receipt checks GET /api/v1/decisions/{sha256}; the exact answer a receipt was made from GET /api/v1/resolve/archive/{sha256}; the answer in force at a past time GET /api/v1/resolve?as_of=; private lookup GET /api/v1/resolve-prefix/{prefix}; site staples GET /api/v1/staple; Resolve over DNS GET /dns-query; the transparency log GET /api/v1/log; the daily signed snapshot GET /api/v1/snapshot; Visitor Resolve GET/POST /api/v1/visitor. The JSON Schema now describes impersonation and transparency. No existing field changed meaning; v stays 1. |
| 1.1.0 | 2026-10-03 | Added: path form GET /v1/resolve/{domain}; batch POST /api/v1/resolve (1-100 domains); change feed GET /api/v1/changes; agent outcome reports POST /api/v1/outcome and GET /api/v1/outcomes; domain lifecycle GET /api/v1/domain; AI accuracy GET /api/v1/accuracy. Answers, change batches, outcome tallies and lifecycle records join the daily sealed Merkle root. No field of ResolveV1 changed. |
| 1.0.0 | 2026-10-02 | First publication: GET /api/v1/resolve, ResolveV1 document with declared / observed / not_measured sections and no overall score, Ed25519 signature, JSON Schema at /schemas/resolve.json, MCP tool resolve_domain, offline verifier. |