When AI and search operators change the addresses their crawlers use.
Operators publish the IP ranges their crawlers come from, so a site can tell the real crawler from an impostor. They change those lists without notice. CrawlCheck re-reads 17 published feeds every day and keeps every change, so a block list, a firewall rule or a log audit can be checked against the list as it stood on the day.
The feeds
Changes
No change recorded yet. Every feed’s first reading is its baseline; the first change appears here the day an operator edits its list.
The full record, with the prefixes added and removed, is at /data/ip-ranges.json (CC BY 4.0). Verify a log line against today’s lists with the free crawler check.