The machine files cost more than the page
A crawler reads robots.txt, the sitemap, llms.txt and the agent files before the first page, and pays for every 404 that answers with a full page. Keep machine files as small as their job allows and serve absent paths with a short 404 body. No site in the newest files has it; the list fills as sites with it are measured.
- 0 sites, last 7 days
- 0 US local businesses
- 0% also block an AI crawler
- $99 once, 30 days of downloads
Who has it
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.