# CrawlCheck

> See what a crawler actually receives. CrawlCheck fetches a site's robots.txt, sitemaps, llms.txt and agents.md from outside the network and reports what a non-browser client receives, including responses that return HTTP 200 while carrying a challenge page instead of the file.

Pages
## Pages

- https://crawlcheck.io/ — Scan any domain and see what each AI crawler was actually served
- https://crawlcheck.io/ai-crawlers — Every named AI crawler we check for, what each is for, and how to verify it
- https://crawlcheck.io/tools — Free checks: can a machine read your files, and is a cache serving something stale
- https://crawlcheck.io/data — Aggregate findings across every site measured here
- https://crawlcheck.io/blog — Measured case studies from real sites
- https://crawlcheck.io/pricing — What the free scan includes and what the two paid tiers add
- https://crawlcheck.io/proof — How every report is fingerprinted and anchored to Bitcoin
- https://crawlcheck.io/policy — What this scanner requests, what it stores, and how to opt out
- https://crawlcheck.io/blog/what-a-crawler-actually-receives — Three failures measured on real local-business sites, none of which a rank tracker, a validator or an uptime check would report.
- https://crawlcheck.io/blog/the-purge-returned-200-and-evicted-nothing — Four successful-looking cache purges cleared zero objects. Here is the two-request test that catches it, and the script that runs it.
- https://crawlcheck.io/blog/a-file-that-answered-200-and-could-not-be-read — A cached challenge page sat where robots.txt should be. Every uptime monitor called it healthy.
- https://crawlcheck.io/blog/which-ai-crawlers-actually-visit-a-small-business-site — Not a probe. Real visits, unasked for, with the operator ranges checked.
- https://crawlcheck.io/blog/one-character-of-undeliverable-email — A contractor site linked the same misspelled address on every page. Two DNS lookups proved it and the fix took one line.


## Machine files
- [entitymap.json](https://crawlcheck.io/entitymap.json): the entity graph for this site
- [agents.md](https://crawlcheck.io/agents.md): instructions for agents reading this site

## What it measures
- Whether the machine files resolve, and whether a 200 actually contains the file
- What each named crawler receives compared with a browser, from one address in one second
- How many of the delivered bytes are visible text
- Whether declared entities, coordinates and profile links resolve to the thing they claim