# CrawlCheck - full technical guide for agents and developers

> CrawlCheck measures what a non-browser client actually receives from a domain: robots.txt, sitemap, llms.txt, agents.md, entitymap.json, and the homepage fetched under fifteen crawler identities in the same second. Every figure in a report comes from a fetch made at scan time.

This file is the technical companion to https://crawlcheck.io/llms.txt. It covers the API, the MCP server, the record format, and how to read a grade honestly.

## Endpoints (the 26 documented; everything else is internal)

- POST https://crawlcheck.io/api/scan - body {"domain":"example.com"} - returns the scan record (JSON). Rate limit 5 per 10 seconds per address; 429 means wait, not failed.
- POST https://crawlcheck.io/api/tool/verify - body {"lines":"<raw access-log lines>"} - each crawler claim adjudicated against the operator's published IP ranges: verified, spoofed, or unverifiable.
- GET https://crawlcheck.io/api/telemetry - verified-crawler traffic observed at this origin.
- GET https://crawlcheck.io/api/corpus/state - dataset coverage by platform, rendering, size, language, kind.
- GET https://crawlcheck.io/api/public/counts - the figures this site quotes about itself; CORS open, cached 60 s.
- GET https://crawlcheck.io/api/registry?host= or ?shelf= - the registry: which measured domains serve an llms.txt, agents.md, media kit, reciprocity-tested entity graph, named AI access policy or agent-callable surfaces.
- GET https://crawlcheck.io/api/globe.json - the cities scans were run from in the last seven days, aggregated; no domain is named.
- GET https://crawlcheck.io/api/tool/robots?domain=&path= - every named crawler resolved against the domain's robots.txt, group shadowing shown.
- GET https://crawlcheck.io/api/tool/llms?domain= - an llms.txt draft from the domain's own pages (&format=txt for the file).
- GET https://crawlcheck.io/api/fix/robots?domain= - the served robots.txt, corrected, every change listed at the top.
- GET https://crawlcheck.io/api/fix/entitymap?domain= - a starter entitymap.json from the latest scan record; needs a prior scan.
- GET https://crawlcheck.io/api/flow?domain=&agent=&fresh= - the observed data flow: where each crawler identity's path dies (edge, robots.txt file, robots.txt rules) and which stores it reached; scans first when no record is on file.
- GET https://crawlcheck.io/api/video?v= | /api/video/channel?handle= | /api/video/site?domain= - YouTube video, channel and site checks.
- Licence-gated for a domain the key covers (x-crawlcheck-key, 404 otherwise): GET /api/flow/gate?domain=&fresh=&since= (CI: 200 pass, 412 fail when an open crawler path closes or a declared path is violated); GET/PUT/DELETE /api/flow/declared?domain= (the owner's declared flow; PUT {"seed":"observed"} to start from the latest scan).
- Licence-gated (x-crawlcheck-key, 404 without): POST /api/scan/licensed, POST /api/depth (result at GET /api/depth/{id}), POST /api/entity/observe.

OpenAPI 3.1: https://crawlcheck.io/openapi.json. Human reference: https://crawlcheck.io/docs/api. Everything else the service answers is internal and may change.

## MCP server

Streamable HTTP, stateless, JSON-RPC 2.0 at POST https://crawlcheck.io/mcp. Card: https://crawlcheck.io/.well-known/mcp/server-card.json. Tools: scan_domain {domain}, verify_crawler_log {lines}, corpus_state {}, resolve_robots {domain, path?}, draft_llms {domain}, fix_robots {domain}, fix_entitymap {domain}, video_check {v, fresh?}, video_channel {handle | id}, video_site {domain, handle?}, crawler_path {domain, agent?, fresh?}, public_counts {}, telemetry {} - one per key-free endpoint. Each tool is a wrapper over the endpoint above with the same rate limit; scan_domain returns a trimmed record plus report_url and record_url.

## Reading a record

- grade: A-F, or null. null means the scan could not be graded - most often refused=true, meaning the homepage and a randomised path that cannot exist answered identically (a bot wall, not a site). Report that as unmeasured. It is never a low score.
- overall: AI visibility 0-100, composed as reach (can a crawler get in), then read (can it parse what came back), then quote (is there a specific fact to cite). Reach is a ceiling: nothing downstream of a refusal reaches an engine.
- section_scores[]: id, title, score. A null score means the section's inputs were not measurable and it is excluded, never counted as zero.
- findings[]: code, severity 1-5, path, title, detail, meaning, evidence. The letter grade is capped by the worst finding (medium -> C, high -> D, critical -> F) and the cap reason is stated in grade_detail.
- self_audit: invariants checked on the record itself. A finding here is a defect in CrawlCheck, not in the site.
- headroom: arithmetic on the failing rows - the points each would add if fixed. Not a forecast.

## Report and record URLs

https://crawlcheck.io/r/{id} renders the report; https://crawlcheck.io/r/{id}.json and .csv export it. Reports regenerate from the stored record, so a link stays true to the measurement it was taken from.

## What this tool does not do

It does not impersonate crawlers to get past a block - being blocked is the measurement. It cannot say whether ChatGPT or Perplexity cites a site; nobody can from the server side. It cannot scan crawlcheck.io itself: a Worker cannot fetch its own route.

## Other machine files on this host

- https://crawlcheck.io/robots.txt
- https://crawlcheck.io/sitemap.xml
- https://crawlcheck.io/llms.txt
- https://crawlcheck.io/agents.md
- https://crawlcheck.io/SKILL.md
- https://crawlcheck.io/entitymap.json
- https://crawlcheck.io/.well-known/api-catalog
- https://crawlcheck.io/.well-known/agent-skills/index.json