How we measure, and our own machine files
CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. What we collect and how to opt out · how every report is timestamped · who runs this · press kit · hello@crawlcheck.io.
Second vantage: the browser extension measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 it also looks up the grade of the site in the current tab by its hostname (never the URL or page content), which a footer switch turns off. Earlier versions send nothing.
Our own machine layer for the agentic web: llms.txt · llms-full.txt · agents.md · SKILL.md · entitymap.json · media kit · MCP server card · API catalog · OpenAPI · robots.txt · sitemap.xml · security.txt — every one named in robots.txt, which is exactly what we check for on yours.