Library · llms.txt, sitemaps and machine files
llms.txt, sitemaps and machine files
In one sentence
Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it; a business found with a defect is named only after its owner has been contacted.
The files written for machines, and whether anything reads them.
2026-11-0212 min
XML sitemap audit: check the file and every URL in it
A sitemap can return 200, open in a browser and still be unusable. The audit has two layers, verify the document and then verify every URL it declares, and most audits stop after the first.
NO_SITEMAP_FOUND10.2%ROBOTS_NO_SITEMAP8.6%DECLARED_SITEMAP_BROKEN0.9%+6
Read the measurements →
2026-11-024 min
Does llms.txt work? What is known and how to test it
No model provider documents it as an input, so the honest answer is a test, not an opinion. We ran that test on our own domain, and you can run it on yours.
NO_LLMS_TXT32.7%ROBOTS_NOT_2005.6%ROBOTS_IS_HTML0.1%+5
Read the measurements →
2026-11-02 · Research5 min
Linked from every page, our llms.txt was fetched 8 times
In two weeks, sixteen AI and search crawler names made 12,831 requests to crawlcheck.io and asked for llms.txt eight times. Per URL, that is the rate of an ordinary page, not a front door.
Read the measurements →
2026-11-025 min
IndexNow: which engines use it and how to check it worked
IndexNow lets a site tell search engines a URL changed instead of waiting to be recrawled. One submission reaches every participating engine. Which engines those are, the key file, the response codes, and what a 200 does and does not mean.
SITEMAP_NO_LASTMOD6.7%
Read the measurements →
2026-11-026 min
How to write an llms.txt and check it is actually served
The file is quick to write. Confirming that the version a crawler receives is the version you wrote is the step almost everyone skips.
ROBOTS_RULES_SHADOWED13%MACHINE_FILE_CACHE_STALE0.5%STALE_CACHE_SERVED15.4%+4
Read the measurements →
2026-11-02 · Case studies6 min
agents.md: AI agent instructions you may not have written
Strip the brand and the domain out of two unrelated agents.md files and they become the same document. Neither owner wrote a word of it.
Read the measurements →
2026-11-024 min
security.txt (RFC 9116): fields, location and expiry rule
security.txt tells someone who found a vulnerability where to report it. RFC 9116 requires two fields and one location. What goes in the file, the expiry rule most files break within a year, and how to check yours from outside.
Read the measurements →
2026-11-024 min
Best llms.txt generators and validators compared
Yoast, Mintlify, Firecrawl and CrawlCheck compared on writing llms.txt, keeping it current, and checking that AI crawlers actually receive it.
NO_LLMS_TXT32.7%MACHINE_FILE_CACHE_STALE0.5%
Read the measurements →
2026-11-02 · Case studies5 min
Sitemap lastmod: 103 pages, one date, why engines skip it
Our sitemap told search engines that 103 pages last changed on 23 September, though most had been rebuilt since. Why Google and Bing ignore dates like that, how we dated each page after the fact, and the rule that keeps it right.
SITEMAP_NO_LASTMOD6.7%
Read the measurements →