CrawlCheck

Datasets · by problem

llms.txt links to pages that do not load

llms.txt is the map an AI agent reads to find your key pages. Some of the links in it answer 404, 410 or a server error, so an agent following the map lands on a dead page or gives up on it. 8,503 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 8,503 sites, measured in the last 7 days
  • 6290 US local businesses
  • 2026-11-20 last counted, recounted every hour
  • $199 once, about 2.3¢ a row, 30 days of downloads

At a glance: 64% of these sites run Wix, and the most common grade among them is C (74%). How CrawlCheck measures

Who has it

Platform

Wix 64%Shopify 16%No CMS detected 8%WordPress 7%Next.js 4%Webflow 1%

Grade

C 74%D 14%A 9%B 3%F 0%

Where

US local businesses 74%Wider corpus 12%SaaS and software 11%Government and education 3%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareOtherANo CMS detectedallowed
SaaS and softwareEcommerceCShopifyallowed
SaaS and softwareOtherBNext.jsallowed
SaaS and softwareOtherDNo CMS detectedallowed
SaaS and softwareSaasAWordPressallowed
SaaS and softwareOtherCElementorallowed
SaaS and softwareSaasCNext.jsblocked
SaaS and softwareOtherCNo CMS detectedallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · free sample of this list · a problem, country or industry we do not list: order a custom dataset.