CrawlCheck

Datasets · by problem

Websites by problem

58 problems we find when we read a site the way search and AI crawlers do. Each list is every site that had the problem when we measured it in the last 7 days, with its platform, grade, location where known and AI-crawler policy: a ready list of businesses that need the fix you sell. $99 up to 1,000 sites, $199 up to 10,000, $299 above. All lanes ($899/month) includes every one.

ProblemSitesMost common platformPrice
No llms.txt84,402No CMS detected 57%$299onceSee the list
HTTPS is served without a Strict-Transport-Security header49,868No CMS detected 56%$299onceSee the list
The homepage is reached through a chain of redirects44,997No CMS detected 46%$299onceSee the list
robots.txt declares no sitemap29,673No CMS detected 66%$299onceSee the list
No XML sitemap found28,053No CMS detected 82%$299onceSee the list
www and the apex both serve the site instead of one redirecting12,448No CMS detected 87%$299onceSee the list
robots.txt does not return 20012,401No CMS detected 82%$299onceSee the list
robots.txt rules do not apply to named agents11,356No CMS detected 44%$299onceSee the list
Links on this site point at pages that do not answer10,737No CMS detected 69%$299onceSee the list
The sitemap does not say when anything changed8,651No CMS detected 61%$199onceSee the list
robots.txt answers with the catch-all page - there is no robots.txt5,187No CMS detected 75%$199onceSee the list
Images are not the format their file names say4,923No CMS detected 62%$199onceSee the list
The sitemap is refused3,604No CMS detected 51%$199onceSee the list
This origin refused CrawlCheck, so the site was never measured3,530No CMS detected 100%$199onceSee the list
robots.txt blocks every crawler it does not name2,580No CMS detected 84%$199onceSee the list
robots.txt keeps Google's image crawler off the site's images2,437No CMS detected 74%$199onceSee the list
The HSTS header asks for browser preloading but does not qualify for it2,058No CMS detected 69%$199onceSee the list
Whether a sitemap exists could not be determined — every path was refused1,347No CMS detected 84%$199onceSee the list
robots.txt points at a sitemap that fails1,333No CMS detected 60%$199onceSee the list
An embedded frame on the homepage cannot load661No CMS detected 58%$99onceSee the list
The origin returned a server error on the sitemap path625No CMS detected 72%$99onceSee the list
robots.txt is unreadable557No CMS detected 74%$99onceSee the list
robots.txt is HTML, not text517No CMS detected 81%$99onceSee the list
The homepage refused this client, so every homepage section is unmeasured516No CMS detected 100%$99onceSee the list
robots.txt has an unexpected content type441No CMS detected 90%$99onceSee the list
The cache is still serving a file the origin no longer has423No CMS detected 64%$99onceSee the list
The sitemap returns HTML, not XML379No CMS detected 66%$99onceSee the list
The instruction files contradict each other377No CMS detected 60%$99onceSee the list
A staging hostname is exposed in the page source346WordPress 84%$99onceSee the list
The sitemap robots.txt declares refused this client188No CMS detected 49%$99onceSee the list
The homepage answered with an interstitial, not a page133No CMS detected 100%$99onceSee the list
CSP violation reporting is misconfigured81No CMS detected 54%$99onceSee the list
A verification page is being served instead of the file79No CMS detected 84%$99onceSee the list
An AI opt-out signal is set77No CMS detected 48%$99onceSee the list
A security layer refused this file — source unconfirmed74No CMS detected 64%$99onceSee the list
This file answers differently depending on whether the cache is hit38WordPress 68%$99onceSee the list
The homepage and robots.txt both refused this client, so the site was never measured5No CMS detected 100%$99onceSee the list
The verification page is cached at the CDN edge2WordPress 100%$99onceSee the list
This domain was removed from the browser HSTS preload list——$99onceSee the list
The page’s content only exists after JavaScript runs——$99onceSee the list
Ads take too much of the page on a phone——$99onceSee the list
A pop-up covers the page on a phone when it loads——$99onceSee the list
The page's date changed but its words did not——$99onceSee the list
Directories list this phone number for other businesses——$99onceSee the list
Pages meant to be found hide their own search snippet——$99onceSee the list
The machine files cost more than the page——$99onceSee the list
Unfinished placeholder text is live on this page——$99onceSee the list
An answer engine's crawler was refused while a browser was served——$99onceSee the list
Almost nothing a machine receives from this page is readable text——$99onceSee the list
Visitors and crawlers are being served an old copy of this page——$99onceSee the list
A named crawler was served materially less text than an ordinary client——$99onceSee the list
A named crawler was sent somewhere the ordinary client was not——$99onceSee the list
This domain is an alias: every path redirects to another site——$99onceSee the list
The homepage sends crawlers to a different hostname——$99onceSee the list
A sitemap answers 200 and lists nothing——$99onceSee the list
Two different entities share the same identifying data——$99onceSee the list
An entity is missing geographic coordinates——$99onceSee the list
Published coordinates are not in the country the address names——$99onceSee the list

38 of 58 problems seen in the newest files; counts are distinct sites measured in the last 7 days, recounted every hour (2026-10-17 14:49 UTC). Sites are named only in the file you buy. Order a custom list.