CrawlCheck

Guides · 2026-10-03 · By · 2 views

Verify GPTBot, ClaudeBot and PerplexityBot: tools compared

Where OpenAI, Anthropic, Perplexity and Google publish their crawler addresses, the tools that check traffic against them (CrawlCheck, Cloudflare, manual), and why unverifiable is its own answer.

To verify an AI crawler, match the request's IP against the operator's published list: OpenAI's gptbot.json, searchbot.json and chatgpt-user.json; Anthropic's ranges for ClaudeBot; Perplexity's perplexitybot.json; Google's reverse DNS or JSON files. CrawlCheck's free log verifier labels pasted log lines verified, spoofed or unverifiable, and Cloudflare's verified-bot signal checks requests at the edge on proxied sites.

Check your own site free

About a minute. First scan free with your email, nothing installed.

Across every full scan CrawlCheck has run, an answer engine's crawler was refused while a browser was served on 5.8% of sites, so verify the refused requests against published ranges before deciding whether a rule is blocking the real crawler.

Anyone can put “GPTBot” in a user-agent header. A log full of GPTBot requests, or of GPTBot 403s, says nothing until each request is checked against the addresses OpenAI actually publishes. The same holds for ClaudeBot, PerplexityBot and Googlebot. This guide lists the published sources, the tools that check against them, and what each one leaves out.

Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing or documentation page, linked where it appears, read on 3 October 2026. Where CrawlCheck does less than a competitor, the tables say so.

Where each company publishes its crawler addresses #

OperatorCrawlersPublished IP list
OpenAIGPTBot, OAI-SearchBot, ChatGPT-Usergptbot.json, searchbot.json, chatgpt-user.json
AnthropicClaudeBot, Claude-SearchBot, Claude-UserPublished; linked from Anthropic’s crawler page
PerplexityPerplexityBot, Perplexity-Userperplexitybot.json, perplexity-user.json
GoogleGooglebot and other crawlersReverse then forward DNS to crawl-*.googlebot.com (never googleusercontent.com, a customer domain), or the JSON range files in Google’s verification guide

Tools that verify crawler traffic #

ToolHow it verifiesWorks onPrice
CrawlCheck log verifierPaste raw access-log lines; each one naming a known crawler is checked against that operator’s published ranges and labelled verified, spoofed or unverifiableAny server’s logsFree
Cloudflare verified botsWeb Bot Auth signatures, published IP lists with a stable user-agent, or reverse DNS; usable in WAF custom rules and AI bot policiesSites proxied through CloudflareAvailable to Cloudflare customers
Manual checkDownload the JSON lists and match IPs yourself, or run reverse and forward DNS for GoogleAnythingFree; your time

Sources: Cloudflare verified bots, the operators’ pages above, and CrawlCheck’s free tools.

Why “unverifiable” is its own answer #

Not every crawler operator publishes addresses (which ones do not). A claim from a crawler with no published list cannot be proven real or fake, and treating it as either distorts the totals. CrawlCheck reports three outcomes and excludes unverifiable claims from every rate, so a share of “forged GPTBot traffic” is computed only over claims that could be checked. The method and the figures are in how much AI crawler traffic is forged.

Step by step: is this GPTBot request real? #

  • Copy the log lines that name GPTBot (or any AI crawler) from your access log.
  • Paste them into the log check on CrawlCheck’s free tools. Each line returns verified, spoofed or unverifiable, with the range it matched.
  • If most “GPTBot” 403s are spoofed, your host is blocking an impostor, not OpenAI; leave the rule. If verified requests are refused, fix the rule that refuses them.
  • On Cloudflare, use the verified-bot signal in rules rather than matching the user-agent string.

Checking one IP against a published list #

OpenAI’s and Perplexity’s files are JSON with a prefixes array of IP ranges in CIDR form, for example {"ipv4Prefix": "132.196.86.0/24"}, plus a creationTime. A request is from the crawler only if its address falls inside one of those ranges. Python’s standard library does the match:

import ipaddress, json, urllib.request
ranges = json.load(urllib.request.urlopen("https://openai.com/gptbot.json"))["prefixes"]
ip = ipaddress.ip_address("132.196.86.25")
print(any(ip in ipaddress.ip_network(r.get("ipv4Prefix") or r.get("ipv6Prefix")) for r in ranges))

Download the list again on a schedule rather than copying it once; the creationTime field changes as ranges are added and retired. Anthropic links its ranges from its crawler page; read that page for the current location and format.

Checking Googlebot with DNS #

Google’s method is two lookups, and both must agree. First a reverse lookup on the address from your log, which for Googlebot must return a crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com name. Google's page also lists google.com (its special-case crawlers) and googleusercontent.com, but the last is its customers' domain: every Google Cloud machine passes a check that accepts it (measured). Then a forward lookup on that name, which must return the original address. Google’s own example:

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The forward lookup is the part people skip, and it is the part that matters: anyone who controls the reverse DNS for their own addresses can make them point at a googlebot.com name, but only Google can make that name resolve back.

Four mistakes that make crawler logs misleading #

  • Blocking by user-agent string alone. It refuses the real crawler and anyone impersonating it alike, and the impersonators simply change the string.
  • Treating unverifiable as fake. A crawler whose operator publishes no list cannot be proven either way; counting it as forged inflates the forged share.
  • A range list copied once. Ranges change; a list from last year will mark new, genuine addresses as spoofed.
  • Ignoring IPv6. If your server logs IPv6 addresses, check them against the IPv6 entries, not only the IPv4 ones.

What to do with each outcome #

OutcomeMeaningAction
Verified and servedThe real crawler read the pageNone
Verified and refused (403, 429, challenge)Your own rule is blocking the real crawlerFind the rule in the firewall, CDN or security plugin and allow the published ranges
SpoofedSomeone else using the crawler’s nameRefusing it is correct; do not loosen the rule
UnverifiableThe operator publishes no listDecide by behaviour (request rate, paths) rather than by name

Where CrawlCheck is weaker #

CrawlCheck checks the log lines you paste; it does not sit in front of your traffic, so it cannot block or rate-limit anything in real time. Cloudflare verifies every request at the edge as it arrives and can act on it, including through Web Bot Auth signatures. For sites on Cloudflare, use both: Cloudflare to enforce, CrawlCheck to audit logs from any host. Background: how to verify Googlebot, Bingbot and AI crawlers by IP address.

Share of scanned sites with each finding in this articleAnswer-engine crawler refused5.8%Share of scanned sites with eachfinding in this articleAnswer-engine crawler refused5.8%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works