What CrawlCheck sends, how a site opts out, and how to report a problem
In one sentence
The security page states what a CrawlCheck request is, how a site opts out of measurement with one robots.txt line, and how to report a vulnerability in crawlcheck.io, its API, the browser extension or the published code.
Two different requests land here, and they are not the same. A vulnerability in crawlcheck.io, its API, the browser extension or the published open-source code is a report: the form below, or an email to hello@crawlcheck.io with the subject line Security report. A site that does not want to be measured opts itself out in its own robots.txt. No email, no form, no account. The machine-readable version of this page is security.txt.
What we send · Opt out · Report a vulnerability · When something goes wrong · Is this IP really Googlebot?
What a CrawlCheck request is
Machine-file requests identify themselves as Mozilla/5.0 (compatible; CrawlCheck/1.0; +https://crawlcheck.io/bot) and are signed. A scan reads public URLs only — robots.txt, sitemaps, llms.txt, llms-full.txt, agents.md, entitymap.json, a media kit, an API catalog, an MCP server card — and never signs in or submits a form. The same files for this site are linked from the footer.
Crawler-identity tests are separate. They send named user-agent strings from CrawlCheck’s own network, never from an operator’s published address range, so a site that verifies crawlers can tell a test apart from the operator. A site that refuses those requests is doing it correctly, and the report says so. The names are on AI crawlers. Every report says what was asked for and what came back; how it is sealed is on proof.
That is what an anonymous non-browser client receives. The browser extension is the second vantage: the same check from your own address, after your JavaScript runs. A block that is really a challenge page is covered on Cloudflare, and how the rules are written is on robots.txt for AI crawlers.
How a site opts out
Either of these stops future measurement, deletes the domain’s stored measurements, and removes it from the crawl queue, the registry and every public count from then on. In the directory the vendor’s name stays, marked opted out with the effective date and no score, so an absence never looks like an editorial exclusion. You declare it in a file you already control. We honour it on the next scan: that scan deletes the stored record, its score history and the corpus records built from them. A total published before then is not recomputed — it is a number, not a list, and it names no one.
A group in robots.txt
User-agent: CrawlCheck
Disallow: /Or one line, anywhere in it
CrawlCheck: noneUser-agent: CrawlCheckBot with Disallow: / is honoured as well. The disallow has to be the whole site. A rule for one path does not opt the domain out, and CrawlCheck: none has to be a line on its own.
A page you request yourself can still be scanned — you may be the one asking — but nothing about it enters the dataset or any count. The report you asked for is kept for its link like every report, and is deleted with the rest of the domain’s records on the owner’s request. The verified owner does that at /manage/data. It ends in a signed deletion record: counts and digests, not the bytes that were removed.
An opt-out does not remove the record’s digest from that day’s sealed proof. The digest is a hash, anchored with every other site measured that day; taking it out would break their proofs, and it cannot be read back into the page that was measured.
A file a buyer already downloaded cannot be recalled. Every opted-out domain is published at /api/datasets/removals with the day it left, as the SHA-256 of the lower-cased domain without www. The domain itself is shown only to a key with dataset access. Buyers delete the matching rows, and re-check at least every 30 days and before each re-use. Counts already published are history and are not recomputed.
Holds and the full timeline are on the policy. Someone who does not control the site, and whose own data appears in a report, asks through removal requests.
Report form
Or send it here: it goes straight into the security queue and you get a reference number. Plain text only; describe attachments or link to them.
Lookalike sites
CrawlCheck is only crawlcheck.io. We are not affiliated with any other site, app or account using the CrawlCheck name. To report a site, email or charge that claims to be us, write to hello@crawlcheck.io.
What to include
The URL or endpoint, the steps to reproduce it, what an attacker gains, and whether you believe any data belonging to someone else was exposed. A proof of concept that stops at the first sign of access is enough.
If your report has to carry a secret (a key, a token, a working exploit), send it as a secret note and paste the link into the form: it is encrypted in your browser and deleted the first time we open it.
In scope
crawlcheck.io and its subpaths, the public API under /api, report and share links, the lead form script /f.js, the browser extension, and the crawlcheck-core repository.
Please do not
Access, change or delete data that is not yours; run denial-of-service or high-volume automated tests; submit forged leads through another business’s form; use social engineering or physical attacks; or test third-party services we use (Cloudflare, Stripe, Google).
What we will do
Acknowledge your report within three business days, tell you whether we can reproduce it, and keep you informed until it is fixed. We will not pursue legal action against good-faith research that follows this page. With your permission we will credit you when the fix ships.
When free scans are abused
Free scans run in steps: each site is fetched at most once every two minutes, there is a daily total, and when that total is being spent fast a free scan asks for a short proof of work before it runs (HTTP 428, code pow_required). A browser solves it by itself in about a second; an agent can solve it from the rule in the response. Nothing about the visitor is stored and no third-party challenge is loaded. The last step pauses free scans entirely. Scans with a licence key are never part of this.
When something goes wrong
Every incident gets a severity, and the severity sets how fast it is answered and who is told.
| Severity | What it means | Answered | Contained | Who is told |
|---|---|---|---|---|
| 1 · Critical | Trust is broken or data is out: a signing key compromised, private reports or licence keys exposed, wrong signed answers served at scale | within 1 hour | within 4 hours | affected licence holders by email within 72 hours of confirming, and a public notice |
| 2 · High | A protection failed or the service is degraded for others: an outside observer compromised, abuse slowing scans, a seal, backup or restore test failing | within 4 hours | within 24 hours | affected licence holders by email if their answers or reports were affected |
| 3 · Medium | One feature wrong or down, no data exposed | within 1 business day | next release | the changelog |
| 4 · Low | Wording, layout, a dead link | — | routine release | — |
A critical incident is closed only when its cause is fixed and a check exists that would catch it again.
How this site is run
HTTPS only with HSTS, a restrictive content security policy, and fetch requests that refuse private, loopback and link-local destinations. A licence key is a credential: never paste one into a public issue, screenshot or support thread. The full picture — retention, email security, outside services — is in the trust center.