Datasets · by problem

Links send crawlers to the same pages under many different addresses

A crawler keeps a queue of every address it has found and still has to fetch (its crawl frontier). Each distinct URL is a separate entry and a separate fetch, so one page linked under several query strings, a session id in every link, “add to cart” and “reply to comment” links, or a calendar that links one month further forever all fill that queue with addresses that teach it nothing. Search engines and AI crawlers spend a limited budget per site; what goes on duplicates is not spent on the pages that matter, and the duplicates compete with each other. Counted from real links on the pages read, not guessed from the sitemap. 438 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

• 438 sites, last 7 days
• 163 US local businesses
• 10% also block an AI crawler
• $99 once, 30 days of downloads

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.