{"feed":"CrawlCheck crawler catalogue changes","source":"https://crawlcheck.io/ai-crawlers","how":"Send the cursor back as ?since= to get only what changed after it.","cursor":11,"since":0,"up_to_date":false,"changes":[{"id":1,"date":"2026-11-13","change":"Amazonbot is listed as a training crawler; Amzn-SearchBot and Amzn-User carry Amazon search and live fetches."},{"id":2,"date":"2026-11-13","change":"Mediapartners-Google, APIs-Google, Google-InspectionTool, AdsBot-Google-Mobile and BingPreview are listed as tools, not search indexes."},{"id":3,"date":"2026-11-13","change":"Yandex, YandexBot, Baiduspider, Seznambot, coccocbot-web and PetalBot are listed with the regional search engines."},{"id":4,"date":"2026-11-13","change":"A retired class holds msnbot, Neevabot, Exabot and Sosospider."},{"id":5,"date":"2026-11-13","change":"Every search and answer crawler record carries the operator page it was read from and what that page says robots.txt does."},{"id":6,"date":"2026-11-13","change":"meta-webindexer is listed as Meta AI's search index crawler; Kagibot as a search index crawler."},{"id":7,"date":"2026-11-13","change":"Magpie-crawler (Brandwatch), the Awario crawlers, VelenPublicWebCrawler (Hunter), Google-CloudVertexBot, SemrushBot-OCOB and Scrapy are listed as tools."},{"id":8,"date":"2026-11-13","change":"Records with no operator page found say so; 30 more crawler names are checked in the long-tail list."},{"id":9,"date":"2026-11-13","change":"Each crawler page has a JSON copy; ready-made robots.txt files, server rules and a mirror of operator-published IP ranges are published."},{"id":10,"date":"2026-11-13","change":"Mediapartners-Google and APIs-Google are read as ignoring the wildcard group, as Google documents for AdsBot; KlaviyoAIBot is listed as a tool in both lists."},{"id":11,"date":"2026-11-13","change":"Cloudflare-AI-Search-External is listed as a tool: the crawler Cloudflare AI Search customers use to index sites outside their own account."}]}