What CrawlCheck collects, and how to opt out

If our crawler reached your server and you want to know what it is, this is the page.

Requests carry the user-agent Mozilla/5.0 (compatible; CrawlCheck/1.0; +https://crawlcheck.io/bot). Every request that fetches your machine files and control paths identifies itself this way. The measurement itself is the one exception: to see what each crawler is served, a scan also asks for your homepage once under each of 12 named crawler user-agents (GPTBot, ClaudeBot, Googlebot and the rest), from our own address. That address is not in any operator’s published range, so a site that verifies crawlers properly will refuse those requests, and the report says so. We never use those strings to get past a block, and we never send them anywhere but the homepage.

Signed, so the name can be checked. Every request we send under that user-agent carries an Ed25519 signature (HTTP Message Signatures, the Web Bot Auth profile) over your hostname, and a Signature-Agent header naming this site. The public keys are at /.well-known/http-message-signatures-directory, and that response is itself signed by every key it lists. What the bot is and how it behaves is at /.well-known/signature-agent-card; key changes are recorded at /.well-known/security-key-rotation.json. We own no IP range — requests leave through Cloudflare’s shared network — so /.well-known/ips.json lists none: an address is never proof a request was ours, the signature is. Check any signed request, ours or anyone’s, with the signature checker.

One exception, stated plainly because it matters: when a report measures what each agent receives, we deliberately send those user-agent strings to your public URL and record what comes back. Those requests come from our own address, which is not in the range those operators publish, so a site that verifies crawlers properly should refuse them. That is a correct result and the report says so. Those probes are sent unsigned: a signature would let a site that verifies Web Bot Auth recognise the probe as us and serve it something the real crawler never receives, and the measurement would stop measuring.

• Public URLs only. We never sign in, never submit a form, and never fetch anything behind authentication.
• robots.txt, sitemap.xml, llms.txt, entitymap.json, agents.md, /.well-known/ucp, and the page requested.
• Two made-up paths per host (/cc-control-<random>.txt at the root, and the same name one directory deep), so a server that answers every address with a page is not mistaken for one that has the file, and a rule that only guards a subdirectory is visible.
• The delivered HTML, response headers and status codes for that page.

• A shareable report at /r/<id>, kept indefinitely: 90 days in the fast store, then from the sealed archive, because receipts, disputes and the daily seal point at it. A report started as unlisted stays out of the registry and every index; a private one opens only for the account that holds the site’s domain claim (the first licence to add the domain, or whoever proves control with a file at /.well-known/crawlcheck-domain.txt) and the operator; listing a domain on a licence is not enough.
• The bytes each crawler identity received, kept as evidence so a proof can be re-checked. They are never republished: anyone can confirm a stored copy still hashes to its SHA-256, but the bytes themselves are not served. Bundles are scanned for keys, tokens and private addresses before any section is served.
• Deletion. The verified owner of a domain can delete everything kept about it, stored bytes included, archive included, at /manage/data; a signed deletion record states what was removed. The daily seal keeps only a hash, which reveals nothing once the record is gone. A robots.txt opt-out hides archived reports at once.
• One row per domain holding the measurements and scores, so a re-scan can show what changed.
• Aggregate counters behind the dataset page.

The rates and counts on the dataset page are aggregates and name no domain. Three published sets do name domains: the census lists the sites in its frame, the daily snapshot carries each measured domain’s signed answer, and the dataset files hold one row per measured site (a 25-row sample is free; the files are licensed). A domain that opts out leaves all three.

Two features read data from Google on your behalf, and only after you sign in and grant them. Neither runs unless you do.

• Search Console. With the webmasters.readonly permission we read clicks, impressions, click-through rate and average position per property per day. Read-only; we cannot change anything in Search Console.
• Business Profile. With the business.manage permission we read the profiles your account manages — name, phone, address, website, hours, category — and their daily calls, direction requests, website clicks, impressions and monthly search keywords. Google grants that permission as read-and-write; we only ever read, and the code that holds the token has no write call in it.

What we keep: a refresh token stored against your licence, the profile fields above, and the daily and monthly series. The token is never shown on any page or API. Disconnect on the licence page deletes the token and every series read with it. Where Google reports a keyword count only as “below a threshold”, we store the threshold and say so; we do not turn it into a number.

If page text is ever sent to a third party for analysis — the content-entity check is designed to use Google’s Natural Language API — that check will be named here before it is switched on, with what is sent and what comes back. As of this page’s date it is not on, and nothing a scan reads leaves this service.

Separate from scanning, and stated because a page listing what we keep should list all of it: this site keeps a short operational record of requests made to it, the way any web server keeps an access log. It holds the last 200 requests and a per-address count that expires after eight days, and it is readable only by us.

• None of it is published. No public page and no public API reads it. Where a visitor address does appear in something public — the crawler figures on the dataset page and at /api/telemetry — it is reduced to a network, a /24 or a /48, never a full address.
• Assets are not recorded, and neither is our own tooling, which identifies itself with a signed internal header; the same header without our signature is ignored and the request is logged like any other.
• Query strings are never stored, so a licence key in a URL cannot end up in that record.

These pages load a small script that measures how fast this site was for you — how long the main content took to appear, how quickly it answered your first interaction, and how much the layout moved — and sends one message when you leave the page. We run it on ourselves because we ask other people to run it, and it would be poor form to publish numbers we had not collected the same way.

• No cookie, no identifier, no profile. Nothing links one page view to another, or to you.
• The path only. Query strings and fragments are stripped before the message is sent, for the same reason they are kept out of the access log above.
• On this site only, the arrival message also carries coarse browser traits — whether the browser reports itself as automated, its time zone, first language, screen size and core count — so automated agents can be told apart from people. They stay with the visit for 30 minutes; only per-burst totals for automated traffic are kept longer, for 90 days.
• Timings, plus a coarse element name — which element was slowest to paint, for instance — with connection type and window width. Readings are merged into counts; individual messages are not kept.
• Blocking it costs you nothing. Any content blocker stops /v.js and every page behaves exactly the same.

Either of these stops future measurement, deletes the domain’s stored measurements, and removes it from the crawl queue, the registry and every public count from then on. In the directory the vendor’s name stays, marked opted out with the effective date and no score, so an absence never looks like an editorial exclusion. No email, no form, no account — you declare it in a file you already control and we honour it on the next scan: that scan deletes the stored record, its score history and the corpus records built from them. A total published before then is not recomputed — it is a number, not a list, and it names no one.

User-agent: CrawlCheck
Disallow: /

CrawlCheck: none

A page you request yourself can still be scanned — you may be the one asking — but nothing about it enters the dataset or any count. The report you asked for is kept for its link like every report, and is deleted with the rest of the domain’s records on the owner’s request.

The directory publishes a score for named companies, which is different from every other measurement here, and it runs under stated rules.

• No paid placement, ever. No vendor is listed for payment, no listing is removed for payment, and no score is edited on request. Nothing a licence unlocks changes a measurement anywhere on this site, and a directory that broke that rule would make every other number here worth less.
• Every score links to its evidence. Each row opens the full report: the status codes, headers and bytes each identity received, on a stated date, under a stated score version.
• A dispute earns a re-scan, not an edit. Tell us what is wrong (or email hello@crawlcheck.io) and the site is measured again; the new reading is published as measured. If our instrument was wrong, the correction is published in the findings, as it has been before.
• The opt-out binds here too. The robots.txt line below removes a domain’s measurements from the dataset and the counters. The directory listing itself remains, marked as opted out with no score, so an absence is visible rather than looking like an oversight.
• Categories are ours and are unscored. They describe what a product is primarily sold as. No pricing is published, because a price we did not measure is a claim we cannot stand behind.