How we measure, and our own machine files
CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. What we collect and how to opt out · how every report is timestamped · who runs this · press kit · hello@crawlcheck.io.
Second vantage: the browser extension measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.
Our own machine layer for the agentic web: llms.txt · llms-full.txt · agents.md · SKILL.md · entitymap.json · media kit · MCP server card · API catalog · OpenAPI · robots.txt · sitemap.xml · security.txt — every one named in robots.txt, which is exactly what we check for on yours.