[Datasets](/datasets) · [by problem](/datasets/problems)

# llms.txt links to pages that do not load

llms.txt is the map an AI agent reads to find your key pages. Some of the links in it answer 404, 410 or a server error, so an agent following the map lands on a dead page or gives up on it. 8,503 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

- **8,503** sites, measured in the last 7 days
- **6290** US local businesses
- **2026-11-20** last counted, recounted every hour
- **$199** once, about 2.3¢ a row, 30 days of downloads

[Buy the list — $199](/credits?item=prob_llms-link-broken#buy)[Free sample of this list (CSV)](/api/datasets/sample?problem=llms-link-broken)[Other problems](/datasets/problems) · [all lanes, $899/month](/datasets#plan-all)

**At a glance:** 64% of these sites run Wix, and the most common grade among them is C (74%). [How CrawlCheck measures](/research)

Who has it

Platform

**Wix** 64% **Shopify** 16% **No CMS detected** 8% **WordPress** 7% **Next.js** 4% **Webflow** 1%

Grade

**C** 74% **D** 14% **A** 9% **B** 3% **F** 0%

Where

**US local businesses** 74% **Wider corpus** 12% **SaaS and software** 11% **Government and education** 3%

| Where | Category | Grade | Platform | AI crawlers |
| --- | --- | --- | --- | --- |
| SaaS and software | Other | A | No CMS detected | allowed |
| SaaS and software | Ecommerce | C | Shopify | allowed |
| SaaS and software | Other | B | Next.js | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Saas | A | WordPress | allowed |
| SaaS and software | Other | C | Elementor | allowed |
| SaaS and software | Saas | C | Next.js | blocked |
| SaaS and software | Other | C | No CMS detected | allowed |

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: [what a row holds](/datasets#fields) · [free sample of this list](/api/datasets/sample?problem=llms-link-broken) · a problem, country or industry we do not list: [order a custom dataset](/datasets#order).

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [Conduct: Adsbot google](/agents/conduct/adsbot-google) | **Guides:** [Robots txt allow list cancels your disallow rules](/guides/robots-txt-allow-list-cancels-your-disallow-rules) | **Glossary:** [anchoring](/glossary/anchoring) · [Authority weighting](/glossary/authority-weighting) · [Transactional query](/glossary/transactional-query) · [ARIA](/glossary/aria) · [self-claim ledger](/glossary/self-claim-ledger) · [OpenSearch description](/glossary/opensearch-description)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
