# agents.md — CrawlCheck

CrawlCheck is a verification layer for the agentic web: it measures what a site actually serves to
named crawlers and agents, decides findings by published rules, and signs the answer. This file
describes the site itself, for any agent reading it.

## What is here
- `/` runs a scan on a public domain and returns a report.
- `/data` publishes what is measured: the daily signed snapshot, the census of the most-visited
  sites (every site in its frame is named), agent conduct, and finding rates.
- `/policy` states what is collected, what is retained, and how to opt a domain out.
- `/ai-crawlers` names every crawler identity this service sends and how to verify one; `/policy` states what the crawler collects.

## The site, by door
- `/for-sites#scanForm` — for site owners: scan free, have the fix list implemented, keep it fixed with a certificate — what an AI agent gets from your site
- `/registry#join` — the registry: every measured domain filed on the shelves it earned — llms.txt, agents.md, media kit, entity graph, AI access policy, agent-callable
- `/pricing#trial` — Free for agents and free to scan; what Watch, Fix, Agency, the API plans and the Data add-on add
- `/pricing#fix` — Free for agents and free to scan; what Watch, Fix, Agency, the API plans and the Data add-on add
- `/tools` — Eight free checks: what an agent is told about a site, machine files, crawler claims, robots.txt per agent, llms.txt draft, compare, video, Web Bot Auth
- `/listing-check` — free check: the business record your site publishes against what directories print, and joining the registry
- `/compare` — compare two sites side by side: what each one delivers to the same crawler identities, graded the same way
- `/extension` — The browser extension: what an agent is served from your own address, joined to the signed reading
- `/video` — YouTube video check: chapters, captions, description, language and topics against what a search result can anchor to
- `/datasets` — Datasets: daily files of measured websites by lane, with a free sample
- `/data#census` — Open data: the daily signed Resolve snapshot, the census of the most-visited sites, agent conduct and finding rates
- `/metrics` — CrawlCheck in numbers: decisions issued to AI agents, API and MCP calls, SDK downloads, false-positive rate on appeal, service objectives
- `/registry` — the registry: every measured domain filed on the shelves it earned — llms.txt, agents.md, media kit, entity graph, AI access policy, agent-callable
- `/directory` — The companies that build for AI agents, measured on their own sites — re-measured daily; dated change ledger; CSV and JSON at /directory.csv and /directory.json
- `/proof` — How every report is fingerprinted and anchored to Bitcoin
- `/docs/quickstart` — Add CrawlCheck preflight to an agent: MCP, curl, Python, OpenAI Agents, Claude tool use, LangChain, Cloudflare Workers
- `/tools/agent-view` — Agent view: the signed answer an agent gets about any domain, laid out for a person — the decision per action and the reason
- `/registry/capabilities` — The capability registry: what measured sites declare AI agents can do, each capability in its invocation policy class, declared beside observed, with a signed certificate
- `/agents/conduct` — How each AI and search agent actually behaves here: robots.txt, rate, forged claims, from verified requests only
- `/verify` — verify a record yourself: download its evidence bundle and check the signature, Merkle path and Bitcoin timestamp in the browser or with one Node file
- `/docs/api` — The API agents call before they act: Resolve, preflight, receipts, Visitor Resolve, MCP tools, every endpoint with an example
- `/spec/resolve` — Resolve protocol: the open, versioned specification of the signed answer an AI agent reads before acting on a domain, with endpoints, signature, verification and change log
- `/sdk` — The typed SDK: install from this site with a pinned integrity hash; bundles the offline verifier
- `/keys` — Signing keys: the current crawler and answer signing key, retired keys, and every key event in a hash-chained log whose head is sealed daily
- `/rules` — The rulebook: every finding CrawlCheck can raise, what it means, its fix and share of scans
- `/conformance/` — Run the suite against your own implementation
- `/how-it-works` — How one answer is made: measured from outside, decided by published rules, signed, and verifiable offline
- `/learn` — learn: guides, measured findings and the glossary, each written from a measurement with its live rate
- `/ai-crawlers` — Every named AI crawler and agent, what each is for, how to verify a request, and how each has behaved
- `/guides` — Every guide published here, and what each one measures.
- `/blog` — Measured case studies from real sites
- `/glossary` — Definitions for every machine-layer term CrawlCheck measures.

## For agents
- `/api/v1/resolve?domain=<name>` returns one signed answer about a domain before you act on it;
  the specification is `/spec/resolve`, the signing keys are at `/keys`, and the answer can be
  verified offline. `/api/v1/outcome` accepts a report of what happened when you acted.
- `POST /api/v1/preflight` with a domain and an action returns allow, warn, require_confirmation
  or block for that one action, with a receipt the visited site can check.
- `/api/v1/visitor?ip=&ua=` says whether a request claiming a named crawler came from the ranges
  its operator publishes: verified, spoofed or unverifiable. `/api/v1/conduct` is the signed record
  of how each agent behaved on this site.
- `/rules` lists every finding CrawlCheck can raise; `/disputes` is where a finding is appealed.
- `/mcp` serves the same measurements as MCP tools; `/openapi.json` describes the API.
- `/feed.json` and `/feed.xml` carry every finding and guide; `/glossary.json` is the glossary.
  Any page is also served as Markdown with `Accept: text/markdown`.

## Reports
Individual reports live at `/r/<id>` and are disallowed in robots.txt and served
with `x-robots-tag: noindex`. They are shareable links, not published pages.

## Contact
Opt-out is self-serve and needs no account: see /policy.