Reference
API
Four endpoints, documented because they are worth relying on. Everything else this service answers is internal, undocumented and free to change without notice.
Read this before you build on it
Unversioned and unstable. There is no /v1/ and no deprecation policy. Field names have changed before and will again — one was renamed the week it shipped because the name was a lie about what it counted. Pin nothing, check for the field you need, and treat a missing key as absent rather than an error.
No authentication, and a rate limit. These are open. /api/scan is capped at five requests every ten seconds per address; over that you get a 429 from the edge, not from us. A scan costs roughly thirty subrequests and takes ten to twenty seconds, so treat it as a slow call, not a lookup.
A GET where a POST is expected returns 404, deliberately. A GET is not a submission, and answering one would invite crawlers to run scans.
The four
POST /api/scan
Runs a full machine-layer audit of one domain and returns the stored record: section scores, findings with evidence, the agent view, machine files, and the self-audit result. This is the same record the report page renders, so the two can never disagree.
Request {"domain":"example.com"} as JSON.
Returns The scan record. grade, overall, section_scores, findings, self_audit. A section that could not be measured scores null, never zero.
POST /api/tool/verify
Adjudicates crawler claims in raw access-log lines against the IP ranges each operator publishes. Nothing else offers this free, and it is the one endpoint here with no equivalent elsewhere.
Request Up to 200 log lines, 20KB, as JSON.
Returns Per line: verified, spoofed or unverifiable — three outcomes, never two. The forged rate divides by verified plus spoofed only, and says so in denominator_note.
GET /api/telemetry
This site’s own verified-crawler traffic: which named agents actually arrived, how many claims were confirmed against operator ranges, and how many were forged.
Request No parameters.
Returns Counts by agent, with the verification denominator stated.
GET /api/corpus/state
Coverage of the measured corpus by platform, rendering mode, size, language and site kind — the population behind every percentile and adoption figure quoted on this site.
Request No parameters.
Returns domains_measured (distinct domains) and dimension_tallies (the sum across dimensions, which counts a domain up to five times). The counting_note field explains the difference, because reading one as the other overstates the corpus fivefold.
What is not here
This service answers a good deal more than four paths. The rest are internal: some are gated, some are POST-only, some exist for one page to talk to another. They are undocumented on purpose — not because they are secret, but because publishing a path is a promise to keep it, and these four are the only ones worth promising.
If you are relying on something not listed here and want it supported, say so at hello@crawlcheck.io and it can be documented properly rather than depended on quietly.