How CrawlCheck reads the web, so the datasets can say what a site actually serves.
CrawlCheck Research is the reading behind every CrawlCheck dataset. It fetches public pages under CrawlCheck’s own name, keeps what it was served, dates every reading and never cites a source it did not read. A person reviews what it publishes. This page is its method, in lines you can check.
138 self-checks run every hour; 0 failing at the last one, 2026-10-27 06:18 UTC.
What it reads
| Source | What is taken from it |
|---|---|
| Public web pages | Fetched as CrawlCheck with a contact address in the user agent, robots.txt honoured, one page at a time. A site that refuses is recorded as refused and never worked around. |
| Machine files | robots.txt, llms.txt, sitemaps and entity maps, read as the content type the site serves. |
| Public archives | Older copies of the same public pages, to see what a site removed. |
| Operator documents | Published crawler documentation, IP-range files and signing keys from the operators themselves. |
| Public registers | Public records, each cited by URL. |
The rules it reads by
- Public sources only. Nothing behind a login, a paywall or a private network. No leak sites, no onion services, no purchased data.
- It names itself. Every read carries the CrawlCheck user agent and a link to this site.
- Crawler identities are labelled as tests. To measure what a site serves GPTBot or ClaudeBot, the CrawlCheck census and scanner send that crawler’s user-agent string from CrawlCheck’s own addresses, never from the operator’s published IP ranges, so a firewall that checks addresses can tell them apart. Every figure built this way says it describes what a client declaring that name received, not what the real crawler received.
- Refusals stand. A robots.txt rule, a 403 or a challenge page ends the read. The refusal is recorded; the page is not fetched another way.
- Every reading is dated and kept with the page it came from, so a figure in a dataset can be traced to the fetch that produced it.
- Aggregates only for anything about people: no addresses, no caller details, no review text attributed to a reviewer.
- A person reviews what goes out. Datasets, census editions and research notes are checked before they are published, and the structured data on each names who is responsible.
What it never cites
Rank-tracking resellers, third-party SEO databases, scraped social feeds, forum rumours and anything it could not fetch itself. If a number has no fetch behind it, it is not in a dataset.
Where its work appears
| Output | Where |
|---|---|
| The measured web and the dataset lists | /datasets and /data |
| What the top sites declare and serve | /radar |
| Crawler catalogue and conduct record | /ai-crawlers and /agents/conduct |
| Daily proof of what was published | /proof |
Report an error
If a reading is wrong, write to hello@crawlcheck.io with the page and the figure. A confirmed error is corrected in the dataset, and the correction is dated.
Built and operated by VSNARY for CrawlCheck. The person responsible for what it publishes is Emmanuel Orta.