Policy

What CrawlCheck collects, and how to opt out

In one sentence

CrawlCheck identifies itself as CrawlCheck/1.0 on every request, reads public URLs only (robots.txt, sitemaps, llms.txt, entitymap.json, agents.md and the requested page), never signs in or submits a form, keeps a shareable report for 90 days and one measurement row per domain, publishes aggregate counts without ever naming a scanned domain, and excludes any domain from the dataset, the crawl queue and the directory on the next scan when its robots.txt carries a CrawlCheck disallow or the line CrawlCheck: none.

If our crawler reached your server and you want to know what it is, this is the page.

Who is fetching

Requests carry the user-agent Mozilla/5.0 (compatible; CrawlCheck/1.0; +https://crawlcheck.io/bot). We identify ourselves on every request and never send a user-agent belonging to Googlebot, GPTBot, ClaudeBot or any other operator in order to gain access.

One exception, stated plainly because it matters: when a report measures what each agent receives, we deliberately send those user-agent strings to your public URL and record what comes back. Those requests come from our own address, which is not in the range those operators publish, so a site that verifies crawlers properly should refuse them. That is a correct result and the report says so.

What we read

Public URLs only. We never sign in, never submit a form, and never fetch anything behind authentication.
robots.txt, sitemap.xml, llms.txt, entitymap.json, agents.md, /.well-known/ucp, and the page requested.
The delivered HTML, response headers and status codes for that page.

What we keep

A shareable report at /r/<id> for 90 days, for whoever ran the scan.
One row per domain holding the measurements and scores, so a re-scan can show what changed.
Aggregate counters behind the dataset page.

The dataset page publishes aggregates only. It has never named a scanned domain and is not going to.

Google accounts you connect

Two features read data from Google on your behalf, and only after you sign in and grant them. Neither runs unless you do.

Search Console. With the webmasters.readonly permission we read clicks, impressions, click-through rate and average position per property per day. Read-only; we cannot change anything in Search Console.
Business Profile. With the business.manage permission we read the profiles your account manages — name, phone, address, website, hours, category — and their daily calls, direction requests, website clicks, impressions and monthly search keywords. Google grants that permission as read-and-write; we only ever read, and the code that holds the token has no write call in it.

What we keep: a refresh token stored against your licence, the profile fields above, and the daily and monthly series. The token is never shown on any page or API. Disconnect on the licence page deletes the token and every series read with it. Where Google reports a keyword count only as “below a threshold”, we store the threshold and say so; we do not turn it into a number.

If page text is ever sent to a third party for analysis — the content-entity check is designed to use Google’s Natural Language API — that check will be named here before it is switched on, with what is sent and what comes back. As of this page’s date it is not on, and nothing a scan reads leaves this service.

Requests to this site

Separate from scanning, and stated because a page listing what we keep should list all of it: this site keeps a short operational record of requests made to it, the way any web server keeps an access log. It holds the last 200 requests and a per-address count that expires after eight days, and it is readable only by us.

None of it is published. No public page and no public API reads it. Where a visitor address does appear in something public — the crawler figures on the dataset page and at /api/telemetry — it is reduced to a network, a /24 or a /48, never a full address.
Assets are not recorded, and neither is anything sent with an x-cc-self header, which is how our own tooling excludes itself.
Query strings are never stored, so a licence key in a URL cannot end up in that record.

Performance timings from this site’s own pages

These pages load a small script that measures how fast this site was for you — how long the main content took to appear, how quickly it answered your first interaction, and how much the layout moved — and sends one message when you leave the page. We run it on ourselves because we ask other people to run it, and it would be poor form to publish numbers we had not collected the same way.

No cookie, no identifier, no profile. Nothing links one page view to another, or to you.
The path only. Query strings and fragments are stripped before the message is sent, for the same reason they are kept out of the access log above.
Timings, plus a coarse element name — which element was slowest to paint, for instance — with connection type and window width. Readings are merged into counts; individual messages are not kept.
Blocking it costs you nothing. Any content blocker stops /v.js and every page behaves exactly the same.

How to opt out

Either of these excludes your domain from the dataset, the crawl queue, the registry and the directory, and from every public count from then on. No email, no form, no account — you declare it in a file you already control and we honour it on the next scan: that scan deletes the stored record, its score history and the corpus records built from them. A total published before then is not recomputed — it is a number, not a list, and it names no one.

User-agent: CrawlCheck
Disallow: /

or a single line anywhere in robots.txt:

CrawlCheck: none

A page you request yourself can still be scanned — you may be the one asking — but nothing about it enters the dataset or any count. The report you asked for is kept for its link, for 90 days, like every report.

The directory

The directory publishes a score for named companies, which is different from every other measurement here, and it runs under stated rules.

No paid placement, ever. No vendor is listed for payment, no listing is removed for payment, and no score is edited on request. Nothing a licence unlocks changes a measurement anywhere on this site, and a directory that broke that rule would make every other number here worth less.
Every score links to its evidence. Each row opens the full report: the status codes, headers and bytes each identity received, on a stated date, under a stated score version.
A dispute earns a re-scan, not an edit. Email hello@crawlcheck.io and the site is measured again; the new reading is published as measured. If our instrument was wrong, the correction is published in the findings, as it has been before.
The opt-out binds here too. The robots.txt line below removes a domain from the dataset, the counters and the directory alike. The listing remains, marked as opted out, so an absence is visible rather than looking like an oversight.
Categories are ours and are unscored. They describe what a product is primarily sold as. No pricing is published, because a price we did not measure is a claim we cannot stand behind.

What we do not do

We do not fetch authenticated or private pages.
We do not sell or share the raw scan records.
We do not name individual domains in anything published.
We do not impersonate another operator’s crawler to get around a block.

Questions about this page

QHow do I opt my domain out of CrawlCheck?

Add User-agent: CrawlCheck with Disallow: / to robots.txt, or the single line CrawlCheck: none anywhere in it. No email or form. The exclusion is honoured on the next scan: that scan deletes the stored record and its history, and the domain leaves the dataset, the registry, the directory and the crawl queue, and adds nothing to any public count after that.

QDoes CrawlCheck pretend to be Googlebot or GPTBot?

Never to gain access. When a report measures what each agent receives, it deliberately sends those user-agent strings from its own address and records what comes back; a site that verifies crawler IPs should refuse them, and the report says that is a correct result.

QWhat is stored about my site?

A report at /r/<id> for 90 days, one row per domain holding the measurements and scores so a re-scan can show what changed, and aggregate counters behind the dataset page. Nothing behind authentication is ever fetched.

QDoes the directory accept payment for placement or removal?

No. No vendor is listed for payment, no listing is removed for payment, and no score is edited on request. A dispute is a re-scan published as measured.

Keep reading

How reports are timestampedEvery measurement is fingerprinted and each day anchored.What the published dataset containsNo domain is ever named in it.How we compare to the crawlers we measureThis scanner is one more named client, and it declares itself.