Datasets · by problem

www and the apex both serve the site instead of one redirecting

Both www and the bare domain answer 200 with the same pages, so every URL on this site exists at two addresses and a crawler fetches the whole site twice. A canonical tag does not prevent this: it is read after the response arrives, so the second fetch has already been spent. Nothing looks broken from a browser, which is why this survives for years. The fix is a 301 from one host to the other so there is one address per page. 12,448 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

• 12,448 sites, last 7 days
• 3,486 US local businesses
• 3% also block an AI crawler
• $299 once, 30 days of downloads

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.