Reference
API
In one sentence
The CrawlCheck API is sixteen documented endpoints, twelve of them open and unauthenticated and three licence-gated: POST /api/scan runs the full scan for a domain and returns the same JSON record the report is rendered from; POST /api/tool/verify adjudicates crawler claims in pasted access-log lines against published IP ranges; GET /api/telemetry and GET /api/corpus/state read the public aggregates, and GET /api/public/counts returns the counters this site quotes about itself — sites_measured and domains_in_corpus are two different quantities, as are sections and sections_scored (a report renders every section; only the scored ones carry weight in the grade, and that figure is derived from the weights map itself so it cannot drift from the scorer), crawler_visits is a rolling window rather than a running total, and every response carries an at timestamp, so any figure published anywhere can be checked against the source rather than trusted. CORS is open and it is cached sixty seconds. Scans are rate-limited to five per ten seconds per address, and a valid licence key on POST /api/scan/licensed gets 60 scans per ten minutes for portfolio runs. An OpenAPI description is at /openapi.json.
The endpoints below are documented because they are worth relying on. Everything else this service answers is internal, undocumented and free to change without notice.
Read this before you build on it
Unversioned and unstable. There is no /v1/ and no deprecation policy. Field names have changed before and will again — one was renamed the week it shipped because the name was a lie about what it counted. Pin nothing, check for the field you need, and treat a missing key as absent rather than an error.
No authentication, and a rate limit. These are open. /api/scan is capped at five requests every ten seconds per address; over that you get a 429 from the edge, not from us. A scan costs roughly thirty subrequests and takes ten to twenty seconds, so treat it as a slow call, not a lookup.
A licensed lane for portfolio runs. The per-address limit is right for a person and wrong for a portfolio. A valid licence key on POST /api/scan/licensed (header x-crawlcheck-key) gets its own allowance of 60 scans per ten minutes per key, enforced here rather than at the edge, so a forged header earns nothing: without a valid key the path is a 404. Same handler, same record.
A GET where a POST is expected returns 404, deliberately. A GET is not a submission, and answering one would invite crawlers to run scans.
The endpoints, and which need a key
POST /api/scan
Runs a full machine-layer audit of one domain and returns the stored record: section scores, findings with evidence, the agent view, machine files, and the self-audit result. This is the same record the report page renders, so the two can never disagree.
Request {"domain":"example.com"} as JSON.
Returns The scan record. grade, overall, section_scores, findings, self_audit. A section that could not be measured scores null, never zero.
POST /api/scan/licensed
The same scan as /api/scan on a lane the anonymous rate limit does not cover. Requires a valid licence key in the x-crawlcheck-key header; answers 404 without one, 429 after 60 scans in ten minutes on that key.
Request {"domain":"example.com"} as JSON, plus the header x-crawlcheck-key: cc_….
Returns Identical to /api/scan.
POST /api/tool/verify
Adjudicates crawler claims in raw access-log lines against the IP ranges each operator publishes. Nothing else offers this free, and it is the one endpoint here with no equivalent elsewhere.
Request Up to 200 log lines, 20KB, as JSON.
Returns Per line: verified, spoofed or unverifiable — three outcomes, never two. The forged rate divides by verified plus spoofed only, and says so in denominator_note.
POST /api/depth
Crawls every URL the site declares (up to 300) and maps the links between them: orphans, undeclared pages, per-page status, canonical, text ratio, JSON-LD, noindex. Licence-gated; 404 without one. Returns 202 with a job id; the result lands at /api/depth/
Request {"domain":"example.com"} as JSON, plus x-crawlcheck-key.
Returns {id, state, url}, then the full record: orphans, undeclared, pages[], medians and stated limits.
GET /api/tool/robots
Resolves every named answer engine, search index and training crawler against a domain's robots.txt the way a crawler does: most-specific group only, longest match, allow wins a tie. A Disallow under * never applies to an agent with its own group, and this is the endpoint that shows it.
Request ?domain=example.com&path=/
Returns Per agent: allowed at the path, allowed_root, whether it is named in the file, and whether its rules are shadowed. Training crawlers are listed, never judged.
GET /api/tool/llms
Drafts an llms.txt from the domain's own homepage and up to 25 declared pages, using their titles and descriptions. A draft, not a publication: the header tells the owner to cut it down.
Request ?domain=example.com, add &format=txt for the file itself.
Returns text plus how many pages were declared, read, and carried their own description.
GET /api/video
One YouTube video against the video anchor model: metadata, description, tags, statistics and topics from the Data API; chapters parsed from description timestamps; captions, transcript and storyboard when the browser extension has observed the video; twelve checks, each naming what it reads. Cached six hours.
Request ?v=<video id or URL>, add &fresh=1 to bypass the cache.
Returns video, chapters, captions, storyboard, thumbnail, checks{rows, passed, scored} and not_measurable.
GET /api/video/channel
Every upload on a channel, read 50 at a time, checked like a single video and tallied per check. Watch and above for the tally; the upload list alone is free.
Request ?handle=@channel or ?id=<channel id>; add &checks=1 for the tally.
Returns rollup{check: {pass, judged, na}} and videos[] sorted by views, each with passed/scored and its failing row ids.
GET /api/video/site
The site half: homepage plus up to 60 sitemap pages read for YouTube embeds, facades and page-builder widgets, VideoObject nodes and their required properties, video spread across pages, and — with a handle — how many of the channel’s videos the site carries.
Request ?domain=example.com, optionally &handle=@channel.
Returns pages[] with embeds and VideoObject counts, videos{} keyed by id, and rows[] with passed/scored.
GET /api/fix/robots
Your served robots.txt, corrected: * group rules copied into every named group that lacked them (shadowing), a Sitemap line added when the file never named the sitemap it serves, a minimal replacement when the served file was HTML. Every change is listed at the top of the file. Nothing else is touched.
Request ?domain=example.com, add &format=json for the notes as data.
Returns text/plain, downloadable as robots.txt.
GET /api/fix/entitymap
A starter entitymap.json built from the latest scan record: name, phone, coordinates and every declared service area as entities with SERVES relations. Fields the page never stated are marked TODO, never guessed. Needs a prior scan.
Request ?domain=example.com
Returns application/json, downloadable as entitymap.json.
POST /api/entity/observe
Files a profile page as your own browser saw it, for the entity corroboration section. The scanner fetches every sameAs target from a datacentre address, and some platforms (Instagram, Facebook, TikTok) answer that address with a challenge page, so the row reads unverifiable. Post the page here and the scanner reads the observation in place of its own fetch - only for rows its own fetch left unverifiable; a live corroborated, unreciprocated or dead result is never overridden - and labels the row owner-browser with the observation's date and digest. Requires a valid licence key and is bound to that licence's domains: it can change rows of reports about sites the licensee holds, never a third party's. The page's own canonical or og:url must name the same host as the url, so a page cannot be filed under another profile's key. Kept 30 days. The browser extension does this in one click.
Request {"url":"https://platform.example/yourprofile","html":"<whole page>"} as JSON (or a form field payload holding it), plus the header x-crawlcheck-key: cc_…. At least 512 bytes of page; the first 200 KB are kept.
Returns key, sha, bytes, at, expires_days and bound_to (the licence's domains). 404 without a valid key; 400 when the canonical disagrees with the url.
GET /api/registry
The registry: every domain this scanner has measured, filed on the shelves it earned — llms.txt, agents.md, a media kit at /.well-known/media-kit.json, an entity graph with at least one sameAs fetched and found pointing back, an AI access policy that names crawlers rather than leaving them to the wildcard, and agent-callable surfaces. A row exists because a fetch produced it. Nothing is submitted, and no payment moves a shelf.
Request ?host= for one domain, or ?shelf= (llms, agents, mediakit, entity, aipolicy, agentapi) with ?limit= and ?cursor=. No key. CORS open.
Returns Shelf counts across the corpus and the matching rows, each with grade, findings, shelf codes and whether the fetch is older than 30 days. Human version: /registry.
GET /api/telemetry
This site’s own verified-crawler traffic: which named agents actually arrived, how many claims were confirmed against operator ranges, and how many were forged.
Request No parameters.
Returns Counts by agent, with the verification denominator stated.
GET /api/corpus/state
Coverage of the measured corpus by platform, rendering mode, size, language and site kind — the population behind every percentile and adoption figure quoted on this site.
Request No parameters.
Returns domains_measured (distinct domains) and dimension_tallies (the sum across dimensions, which counts a domain up to five times). The counting_note field explains the difference, because reading one as the other overstates the corpus fivefold.
MCP server and OpenAPI
The endpoints above are reachable two more ways, for clients that prefer them. Neither adds capability; both are second doors to the same rooms, with the same rate limit. MCP offers only the key-free ones — a stateless JSON-RPC call carries no licence.
MCP. A stateless Streamable HTTP server at POST /mcp speaking JSON-RPC 2.0 — no session, no SSE. Tools: scan_domain {domain}, verify_crawler_log {lines}, corpus_state {}, resolve_robots {domain, path?}, draft_llms {domain}, fix_robots {domain}, fix_entitymap {domain}, video_check {v, fresh?}, video_channel {handle | id}, video_site {domain, handle?}, public_counts {}, telemetry {}. scan_domain returns a trimmed record (grade, score, sections, findings, self-audit) plus report_url and record_url for the full one. Card: /.well-known/mcp/server-card.json.
Not ours: the npm package named crawlcheck and the GitHub project crawlcheck-mcp-server are an unrelated Playwright crawler by a different author that reports console errors and broken links. CrawlCheck publishes no npm package. The only MCP surface for this service is POST /mcp on crawlcheck.io, described by the server card above.
OpenAPI 3.1. /openapi.json describes every endpoint above and nothing else; the build harness asserts its path set equals this page's card set, so the two cannot drift. Also linked as service-desc from /.well-known/api-catalog.
For agents reading this as text: /llms-full.txt carries the same reference plus how to read a record — in particular that a null grade means unmeasured, never a low score.
What is not here
This service answers a good deal more than the paths above. The rest are internal: some are gated, some are POST-only, some exist for one page to talk to another. They are undocumented on purpose — not because they are secret, but because publishing a path is a promise to keep it, and these are the only ones worth promising.
If you are relying on something not listed here and want it supported, say so at hello@crawlcheck.io and it can be documented properly rather than depended on quietly.