Datasets · by problem

Links on this site point at pages that do not answer

A link on this site was followed and the target answered 4xx or 5xx. A crawler spends budget on every one of these and learns nothing; a reader following one gets an error page. The usual causes are a route renamed without updating the links to it, and a nav or footer that renders links to authenticated-only routes for everyone — those answer 404 when signed out, which is what a crawler always is. Only 404 and 410 count here. A link answering 401, 403 or 429 means something declined to answer CrawlCheck — bot protection, a login wall, a rate limit — which is a fact about the request, not about the link, so those are counted and never reported as broken. Links are sampled rather than exhaustively crawled, so this is a floor and not a full link audit. 10,737 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

• 10,737 sites, last 7 days
• 3,381 US local businesses
• 5% also block an AI crawler
• $299 once, 30 days of downloads

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.