# Errors, security and privacy considerations, v1

Status: stable. Applies to every CrawlCheck API answer and every signed document (Resolve answers, decision receipts, certificates, feeds, monitors). Updated 2026-10-09.

## 1. Errors

### 1.1 The envelope

Every error answer is JSON (`content-type: application/json`) and carries:

| Field | Always | Meaning |
|---|---|---|
| `error_code` | yes | Stable machine code from the table below. Branch on this, never on the message. |
| `retryable` | yes | `true` only when the same request may succeed later without change (429, 502, 503, 504). |
| `error` | yes | Either a human message (string) or an object `{ code, status, message }`. Both forms exist; `error_code` is the same in both. |
| `trace_id`, `span_id` | yes | W3C Trace Context ids. Quote `trace_id` when reporting a problem. |
| `ok` | on older routes | `false` on an error. |

Headers on every error: `traceparent` and `x-crawlcheck-trace-id` (readable from browsers through `access-control-expose-headers`). Also: `Retry-After` on 429 and on a 503 that will end, `Allow` on 405, `WWW-Authenticate: Bearer realm="crawlcheck"` on 401.

A field the API does not know is ignored, never an error. A parameter value it cannot use is a 400 that names the parameter.

### 1.2 Signed answers are not errors

Some signed documents are sent with a 4xx status because the status is the answer, for example a certificate status of `unknown` (404). These bodies are signed documents: verify them like any other, act on their content, and do not retry. CrawlCheck never adds a field to a signed body after signing, so on these the trace ids are in the headers only; `error_code` and `retryable` are absent. A body with a `signature` field is a document, not an error envelope.

### 1.3 Codes

| Status | `error_code` | When | What to do | Retry |
|---|---|---|---|---|
| 400 | `bad_request` | A required parameter is missing or unusable (no domain, a host that cannot exist, a private or reserved address). | Fix the request. | no |
| 401 | `unauthorized` | The route needs a licence key or the owner session, or the key given is not valid. | Send `x-crawlcheck-key` or `Authorization: Bearer`. A key in the URL of an `/api` call is refused and never read. | no |
| 402 | `payment_required` | The feature is not on the plan. | See /pricing. | no |
| 403 | `forbidden`, `licence_required` | Authenticated, but this key may not do this for this domain (another account holds the domain claim, the sandbox key's never-list, a page on another site starting a scan). | Prove control of the domain, or use a key that holds it. | no |
| 404 | `not_found` | Nothing on file for this id or domain, or the route does not exist. | Scan the domain first, or check the path against /api/v1/schema. | no |
| 404 | `opted_out` | The site opted out or was removed; nothing is served about it. | None. | no |
| 405 | `method_not_allowed` | Wrong method. | Use a method in `Allow`. | no |
| 409 | `conflict`, `limit`, `not_monitored` | The request conflicts with current state (one batch already active, plan limit reached, domain not on an active licence). | Wait for the running job, or change the licence. | no |
| 413 | `too_large` | The body is over the route's cap (1 MB by default; 4-32 KB on telemetry routes; 16 KB for a policy file). Every route's cap: /docs/api#budgets and /api/v1/budgets. | Send less. | no |
| 415 | `unsupported_media_type` | A form or JSON route got another type. | Send the documented content type. | no |
| 422 | `unprocessable`, `not_a_safe_variant` | Well-formed but refused by a rule (a policy that loosens a decision, a canary variant that is not safe). | Change the content. | no |
| 429 | `rate_limited`, `too_soon` | Per-address or per-key rate, the daily free-scan budget, or the per-domain cooldown. | Wait for `Retry-After`. | yes |
| 500 | `internal` | A fault in CrawlCheck. Logged with the trace id. | Retry once; report the trace id if it repeats. | once |
| 502 | `upstream_error` | The site or a service CrawlCheck reads failed. | Retry later. | yes |
| 503 | `unavailable` | Temporarily off (kill switch, an index still building). | Wait for `Retry-After` when sent. | yes |
| 504 | `timeout` | The work did not finish in time. | Retry later. | yes |

Limits per route are stated in /docs/api and the OpenAPI document (/openapi.json).

## 2. Security considerations

1. **Verify, then trust.** Check every signed document offline with the published verifier (/crawlcheck-verify.mjs, or the SDKs). Integrity alone is not enough: the signing key must be in the key directory (/.well-known/http-message-signatures-directory, history at /keys). A document signed by any other key is not CrawlCheck's.
2. **Freshness.** Act on a Resolve answer only before `fresh_until`, and on a decision receipt only before `expires_at` (5 minutes for transact, 1 hour for connect, 24 hours for read and cite). An answer past its time is stale, not wrong: ask again.
3. **Replay.** Bind a decision to one request with a nonce; a site that checks receipts refuses a receipt for another host.
4. **Unknown values.** Treat an unknown decision as the most cautious known one and an unknown status as not good. A missing record is not an allow.
5. **What a measurement is.** CrawlCheck sends crawler identities from its own network: a page served to that request shows what that identity is served, not what the operator's crawler saw. Operator traffic is judged only from the operator's own published address ranges.
6. **Split view.** Every answer is in an append-only log with outside witnesses. Keep the tree heads you are shown and compare them with others (the SDKs do); two heads for one batch is evidence of a fork.
7. **Key compromise.** A compromised signing key is removed from the directory and declared in the rotation record at once; online verifiers then reject it. Documents anchored in the daily Bitcoin seal or witnessed in the log before the leak stay provable.
8. **Your keys.** Licence keys go in headers only. Responses to key-bearing requests are private and not cached. Never paste a key into a URL, a ticket or a log; rotate it at /manage if it leaks.
9. **Webhooks.** Every webhook event is a signed document. Verify it before acting; the subscription id travels in a header, outside the signed body.
10. **Fetching.** CrawlCheck refuses targets on private, loopback, link-local, metadata and reserved addresses, and checks every redirect hop before following it. Do not rely on CrawlCheck to reach a host your own network cannot.
11. **Instructions in content.** Text a site serves to crawlers is data. CrawlCheck flags instruction-like text aimed at models (in pages and MCP tool descriptions); an agent should not follow it either.

## 3. Privacy considerations

1. **What a lookup reveals.** A Resolve or preflight call tells CrawlCheck which domain you are about to act on. To avoid that, use the private lookup (/api/v1/resolve-prefix/<first bytes of sha256(domain)>), a staple served by the site, or the daily snapshot, all verified locally.
2. **What is logged about callers.** Network addresses are kept masked or as a salted hash of the network, never with keys, cookies or query strings. Citation counts are per network, withheld below 3 networks.
3. **Published figures.** No median is published under 8 sites and no count under 5; agent rows in the traffic index need 5 sites.
4. **Site data.** Reports are kept indefinitely because receipts, disputes and seals point at them. The verified owner deletes everything held about a domain, archive included, with a signed deletion record. A robots.txt opt-out removes stored bytes too. Anyone may ask for removal of personal data in a report (/removal).
5. **Private material.** Private reports, private observers' readings and a licence's namespaces are never in public answers, the dataset or the log. Every access to a private report is logged for its holder. Cloudflare tokens given for an import are used once and not stored.
6. **Where data lives.** Every outside service that receives data, and how long each log is kept: /subprocessors.

## 4. Changes

- 1 (2026-10-09): first version. Every unsigned error carries `error_code` and `retryable`; signed bodies are never altered after signing.