# CrawlCheck rights feed, v1

What the most-visited sites declare about AI use, one signed line per site, built for crawler operators and AI labs to load before they fetch.

## Files

- `GET https://crawlcheck.io/api/v1/rights-feed` — index: the latest finished edition, the one being built, totals per use.
- `GET https://crawlcheck.io/api/v1/rights-feed/{edition}/manifest.json` — signed (kind `crawlcheck-rights-feed`): every part with its record count and SHA-256.
- `GET https://crawlcheck.io/api/v1/rights-feed/{edition}/part-NNNN.ndjson.gz` — gzip, one JSON object per line, 1,000 sites per part.

Sample edition: the top 10,000 registrable domains of one Tranco list (frozen; sites that opted out of CrawlCheck are removed), rebuilt weekly.

## One line

| field | meaning |
|---|---|
| `domain`, `rank`, `read_at` | the site, its Tranco rank, when CrawlCheck read it |
| `robots` | `url`, HTTP `status`, `readable`, and `groups[]`: each group's `agents`, `allow`, `disallow` and any `content_signal` lines, as written |
| `page` | homepage status, `meta_robots` directives, `x_robots_tag` header |
| `tdm` | TDM-Rep reservation (header, meta tag or /.well-known/tdmrep.json) |
| `ai_txt` | /ai.txt decision for `/` |
| `terms` | the terms page URL and up to two AI or text-and-data-mining clauses, quoted |
| `uses_at_root` | for `/`: `train`, `ground`, `summarize`, `cite`, `transact` → `value` (allowed, denied, limited, restricted_by_terms, not_declared, unknown), the `signal` and `evidence` that decided it, `conflicts` count, and `closed_to` when robots.txt treats agents differently |
| `sha256`, `signature` | Ed25519 over the canonical JSON (kind `crawlcheck-rights`) |

## Applying it to a URL

1. Pick the groups for your product token as RFC 9309 says: the groups that name your token, merged; if none, the `*` groups.
2. Longest matching rule wins; on a tie, Allow wins. `*` matches any run of characters, a trailing `$` anchors the end.
3. Then the use signals: a `Content-Signal` value in your group (`ai-train`, `search`, `ai-input`), TDM-Rep, ai.txt and page directives. A denial in any of them denies.
4. `unknown` means CrawlCheck was refused or could not reach robots.txt; it is not permission.

## Proof

Keep the line you acted on. Its signature proves what the site declared at `read_at`, verifiable offline with https://crawlcheck.io/crawlcheck-verify.mjs, and the manifest proves the line was part of that edition. For a single URL at fetch time, `GET /api/v1/rights?domain=&path=&agent=` returns a signed answer for that path and token.

## Limits

A declaration read once a week from one vantage. It is not legal advice, and it does not say what any crawler did.