The sitemap does not say when anything changed
The sitemap lists URLs but carries no lastmod dates, so it says what exists and not what changed. A crawler with a budget uses lastmod to decide what to re-fetch; without it, the choice is made for you — usually by re-crawling little and late. This is the change signal every crawler already understands, and unlike a push protocol it needs no key, no account and no participation from the search engine. 8,651 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.
- 8,651 sites, last 7 days
- 2,964 US local businesses
- 5% also block an AI crawler
- $199 once, 30 days of downloads
Who has it
Platform
No CMS detected 61%WordPress 25%Next.js 8%Shopify 4%Webflow 2%Drupal 1%
Grade
C 53%A 22%B 11%D 11%F 3%
Where
Wider corpus 40%US local businesses 34%SaaS and software 16%Government and education 9%
| Where | Category | Grade | Platform | AI crawlers |
|---|---|---|---|---|
| SaaS and software | Finance | A | No CMS detected | allowed |
| SaaS and software | Other | A | No CMS detected | allowed |
| SaaS and software | Saas | A | No CMS detected | allowed |
| SaaS and software | Saas | B | No CMS detected | allowed |
| SaaS and software | Saas | A | No CMS detected | allowed |
| SaaS and software | Saas | C | No CMS detected | allowed |
| SaaS and software | Other | A | No CMS detected | allowed |
| SaaS and software | Other | C | No CMS detected | allowed |
A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.