Datasets · by problem

robots.txt rules do not apply to named agents

A crawler reads only the User-agent group that matches it best and ignores every other group, including `*`. Naming an agent and giving it only `Allow: /` therefore deletes all of your `*` Disallow rules for that agent. The file still parses and the agent still reaches your homepage, so this is invisible on inspection — but the paths you meant to keep out of search are open, and search engines will crawl and may index them. 11,356 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

• 11,356 sites, last 7 days
• 5,597 US local businesses
• 14% also block an AI crawler
• $299 once, 30 days of downloads

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.