What CrawlCheck collects, and how to opt out

If our crawler reached your server and you want to know what it is, this is the page.

Who is fetching

Requests carry the user-agent Mozilla/5.0 (compatible; CrawlCheck/1.0; +https://crawlcheck.io/bot). We identify ourselves on every request and never send a user-agent belonging to Googlebot, GPTBot, ClaudeBot or any other operator in order to gain access.

One exception, stated plainly because it matters: when a report measures what each agent receives, we deliberately send those user-agent strings to your public URL and record what comes back. Those requests come from our own address, which is not in the range those operators publish, so a site that verifies crawlers properly should refuse them. That is a correct result and the report says so.

What we read

• Public URLs only. We never sign in, never submit a form, and never fetch anything behind authentication.
• robots.txt, sitemap.xml, llms.txt, entitymap.json, agents.md, /.well-known/ucp, and the page requested.
• The delivered HTML, response headers and status codes for that page.

What we keep

• A shareable report at /r/<id> for 90 days, for whoever ran the scan.
• One row per domain holding the measurements and scores, so a re-scan can show what changed.
• Aggregate counters behind the dataset page.

The dataset page publishes aggregates only. It has never named a scanned domain and is not going to.

How to opt out

Either of these excludes your domain from the dataset, the public counters and the crawl queue. No email, no form, no account \u2014 you declare it in a file you already control and we honour it on the next scan.

User-agent: CrawlCheck
Disallow: /

or a single line anywhere in robots.txt:

CrawlCheck: none

A page you request yourself can still be scanned \u2014 you may be the one asking \u2014 but nothing about it is stored or counted.

What we do not do

• We do not fetch authenticated or private pages.
• We do not sell or share the raw scan records.
• We do not name individual domains in anything published.
• We do not impersonate another operator\u2019s crawler to get around a block.