--- name: crawlcheck description: | The verification layer for the agentic web. Before an agent reads, cites, connects to or transacts with a site, get one signed, offline-verifiable answer about it: what each named crawler was really served, what the site declares (robots.txt, sitemap, llms.txt, agents.md, entitymap.json, MCP and API surfaces) and a decision per action. Also scans any public domain and returns a graded A-F report with evidence. --- # CrawlCheck The verification layer for the agentic web. Every figure comes from a fetch made at measurement time; nothing is asserted from a cache or a model, and every answer is signed with a published Ed25519 key. ## Before you act on a site GET https://crawlcheck.io/api/v1/resolve?domain=example.com One signed answer: per-action decisions (read, cite, connect, transact), freshness and evidence links. Specification at /spec/resolve, keys at /keys; verify offline with /crawlcheck-verify.mjs. No key needed. POST https://crawlcheck.io/api/v1/preflight Content-Type: application/json { "domain": "example.com", "action": "transact" } Returns allow, warn, require_confirmation or block for that one action, with a receipt the visited site can check. ## Scan a domain POST https://crawlcheck.io/api/scan Content-Type: application/json { "domain": "example.com" } Returns a JSON record: `grade`, `overall`, `section_scores`, `findings` (each with a code, severity, path and evidence), and the raw measurements behind them. A full URL is accepted in place of a bare domain. Free scans have no daily limit. Requests are rate limited to 5 per 10 seconds per address; a 429 means that burst limit - wait a minute, or POST to https://crawlcheck.io/api/scan/licensed with a licence key in the `x-crawlcheck-key` header. ## MCP Stateless Streamable HTTP server at POST https://crawlcheck.io/mcp with the tools scan_domain, verify_crawler_log, corpus_state, resolve_robots, draft_llms, fix_robots, fix_entitymap, video_check, video_channel, video_site, registry_lookup, crawler_path, public_counts, telemetry, fix_mostly_code, fix_stale_cache, fix_machine_chain, explain_finding, list_verified_capabilities, machine_record, resolve_domain, preflight and mcp_servers - one per key-free endpoint. Card at /.well-known/mcp/server-card.json. OpenAPI 3.1 at /openapi.json. ## Adjudicate crawler claims in a log POST https://crawlcheck.io/api/tool/verify Paste up to 200 raw access-log lines. Each line naming a crawler is checked against that operator's published IP ranges and returned as **verified**, **spoofed**, or **unverifiable** when the operator publishes no ranges. The forged rate divides by verified plus spoofed only, and the response states that denominator. ## Read this host's own crawler traffic GET https://crawlcheck.io/api/telemetry Verified crawler hits on this site, by agent. ## Corpus coverage GET https://crawlcheck.io/api/corpus/state How many distinct domains have been classified, and where the coverage gaps are. `domains_measured` is the distinct-domain count; `dimension_tallies` counts a domain once per dimension and is not a count of sites. ## File a profile page you observed POST https://crawlcheck.io/api/entity/observe Header: x-crawlcheck-key: cc_... (a licence key; the route is 404 without one) { "url": "https://platform.example/yourprofile", "html": "" } Some platforms answer a datacentre fetch with a challenge page, so the entity corroboration row for that sameAs target reads unverifiable. Post the page as your browser saw it (at least 512 bytes) and CrawlCheck reads the observation in place of its own fetch - only for rows its own fetch left unverifiable, never overriding a live result - and labels the row owner-browser. The observation is bound to the licence's domains and can only change reports about sites the licensee holds. The page's canonical must name the same host as the url. Kept 30 days. Returns `key`, `sha`, `bytes`, `at`, `expires_days` and `bound_to`. ## Registry GET https://crawlcheck.io/api/registry?host=example.com Whether a domain is listed and which shelves it is on: llms, agents, mediakit, entity, aipolicy, agentapi. A shelf is a measurement from the domain's last scan, never a submission. Without `host`: shelf counts and rows, up to `limit` (500) per page from `offset`; `next` is the URL of the following page and is null on the last one. ## Live counters GET https://crawlcheck.io/api/public/counts The figures this site quotes about itself, as JSON, so a number published anywhere can be checked against the source rather than trusted. Returns `sites_measured` and `domains_in_corpus` - two different quantities, which is worth saying plainly because prose on this site has not always distinguished them - plus `crawler_visits`, `queue`, `identities_sent`, `sections` and `sections_scored` (a report renders every section; only the scored ones carry weight in the grade), `findings_published`, `guides_published`, and an `at` timestamp for the moment the counts were read. CORS is open and no key is needed. Cached 60 seconds; safe to poll. Prefer this to scraping /data. `crawler_visits` is a rolling window, not a running total, so it moves in both directions. Read the `at` field before comparing two readings. ## What this cannot tell you It is not a site-wide crawler, not a log platform, and not a Search Console replacement. It measures one rendering of the pages it fetches and does not execute JavaScript. Our vantage point is a datacentre address, which is not the same address a crawler uses - where that matters, the report says so. ## Rules - Reports at `/r/` are disallowed in robots.txt and served noindex. They are shareable links, not published pages. - Opt a domain out at https://crawlcheck.io/policy - self-serve, no account. - Requests under this service's own name say CrawlCheck/1.0 and are signed; the homepage is also fetched once under each named crawler user-agent (the measurement). Both are described at https://crawlcheck.io/policy.