CrawlCheck

Agent view · Guides · 2026-10-03

Verify GPTBot, ClaudeBot and PerplexityBot: tools compared

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/how-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/how-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes16,186
words2,114
sha-256ea1ef481905e4bfeb4c7db61b80ec21f09a2e1eda0ef49bbdbf3b6ccf1c83a29
repr-digestsha-256=:6h70gZBeS/60x9thuA7CHwmi4e2g70m72/O2zPHIOik=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311492;expires=1795315092;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="wPH3YL506mFNz0yydack8w";tag="crawlcheck-page"
signaturesig1=:99rXMwFCDy6N3McLyysuSuhT+BOLyzZY8RCJGcCnMWl17Cind2K2dH/ssY7A+TfmfgGubepX1++b5W/k83AfCw==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-10-03 · By [VSNARY | Emmanuel Orta](/author) · 2 views

# Verify GPTBot, ClaudeBot and PerplexityBot: tools compared

Where OpenAI, Anthropic, Perplexity and Google publish their crawler addresses, the tools that check traffic against them (CrawlCheck, Cloudflare, manual), and why unverifiable is its own answer.

To verify an AI crawler, match the request's IP against the operator's published list: OpenAI's gptbot.json, searchbot.json and chatgpt-user.json; Anthropic's ranges for ClaudeBot; Perplexity's perplexitybot.json; Google's reverse DNS or JSON files. CrawlCheck's free log verifier labels pasted log lines verified, spoofed or unverifiable, and Cloudflare's verified-bot signal checks requests at the edge on proxied sites.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- Where each company publishes its crawler addresses
- Tools that verify crawler traffic
- Why “unverifiable” is its own answer
- Step by step: is this GPTBot request real?
- Checking one IP against a published list
- Checking Googlebot with DNS
- Four mistakes that make crawler logs misleading
- What to do with each outcome
- Where CrawlCheck is weaker

Across every full scan CrawlCheck has run, an [answer engine](/glossary/answer-engine)'s crawler was refused while a browser was served on 5.8% of sites, so verify the refused requests against published ranges before deciding whether a rule is blocking the real crawler.

Anyone can put “GPTBot” in a user-agent header. A log full of GPTBot requests, or of GPTBot 403s, says nothing until each request is checked against the addresses OpenAI actually publishes. The same holds for ClaudeBot, PerplexityBot and Googlebot. This guide lists the published sources, the tools that check against them, and what each one leaves out.

**Disclosure:** CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing or documentation page, linked where it appears, read on 3 October 2026. Where CrawlCheck does less than a competitor, the tables say so.

## Where each company publishes its crawler addresses #

| Operator | Crawlers | Published IP list |
| --- | --- | --- |
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User | [gptbot.json](https://openai.com/gptbot.json), [searchbot.json](https://openai.com/searchbot.json), [chatgpt-user.json](https://openai.com/chatgpt-user.json) |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User | Published; linked from [Anthropic’s crawler page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Perplexity | PerplexityBot, Perplexity-User | [perplexitybot.json](https://www.perplexity.com/perplexitybot.json), [perplexity-user.json](https://www.perplexity.com/perplexity-user.json) |
| Google | Googlebot and other crawlers | Reverse then forward DNS to crawl-*.googlebot.com (never googleusercontent.com, a customer domain), or the JSON range files in [Google’s verification guide](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) |

## Tools that verify crawler traffic #

| Tool | How it verifies | Works on | Price |
| --- | --- | --- | --- |
| **CrawlCheck log verifier** | Paste raw access-log lines; each one naming a known crawler is checked against that operator’s published ranges and labelled verified, spoofed or [unverifiable](/glossary/unverifiable) | Any server’s logs | Free |
| Cloudflare verified bots | [Web Bot Auth](/glossary/web-bot-auth) signatures, published IP lists with a stable user-agent, or reverse DNS; usable in [WAF](/glossary/waf) custom rules and AI bot policies | Sites proxied through Cloudflare | Available to Cloudflare customers |
| Manual check | Download the JSON lists and match IPs yourself, or run reverse and forward DNS for Google | Anything | Free; your time |

Sources: [Cloudflare verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), the operators’ pages above, and CrawlCheck’s [free tools](/tools).

## Why “unverifiable” is its own answer #

Not every crawler operator publishes addresses ([which ones do not](/blog/the-crawlers-nobody-can-check)). A claim from a crawler with no published list cannot be proven real or fake, and treating it as either distorts the totals. CrawlCheck reports three outcomes and excludes unverifiable claims from every rate, so a share of “forged GPTBot traffic” is computed only over claims that could be checked. The method and the figures are in [how much AI crawler traffic is forged](/blog/how-much-ai-crawler-traffic-is-forged).

## Step by step: is this GPTBot request real? #

- Copy the log lines that name GPTBot (or any AI crawler) from your access log.
- Paste them into the log check on [CrawlCheck’s free tools](/tools). Each line returns verified, spoofed or unverifiable, with the range it matched.
- If most “GPTBot” 403s are spoofed, your host is blocking an impostor, not OpenAI; leave the rule. If verified requests are refused, fix the rule that refuses them.
- On Cloudflare, use the verified-bot signal in rules rather than matching the user-agent string.

## Checking one IP against a published list #

OpenAI’s and Perplexity’s files are JSON with a `prefixes` array of IP ranges in CIDR form, for example `{"ipv4Prefix": "132.196.86.0/24"}`, plus a `creationTime`. A request is from the crawler only if its address falls inside one of those ranges. Python’s standard library does the match:

```
import ipaddress, json, urllib.request
ranges = json.load(urllib.request.urlopen("https://openai.com/gptbot.json"))["prefixes"]
ip = ipaddress.ip_address("132.196.86.25")
print(any(ip in ipaddress.ip_network(r.get("ipv4Prefix") or r.get("ipv6Prefix")) for r in ranges))
```

Download the list again on a schedule rather than copying it once; the `creationTime` field changes as ranges are added and retired. Anthropic links its ranges from its crawler page; read that page for the current location and format.

## Checking Googlebot with DNS #

Google’s method is two lookups, and both must agree. First a reverse lookup on the address from your log, which for Googlebot must return a `crawl-*.googlebot.com` or `geo-crawl-*.geo.googlebot.com` name. Google's page also lists `google.com` (its special-case crawlers) and `googleusercontent.com`, but the last is its customers' domain: every Google Cloud machine passes a check that accepts it ([measured](/blog/googles-googlebot-verification-rule-passes-any-google-cloud-server)). Then a forward lookup on that name, which must return the original address. Google’s own example:

```
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
```

The forward lookup is the part people skip, and it is the part that matters: anyone who controls the reverse DNS for their own addresses can make them point at a googlebot.com name, but only Google can make that name resolve back.

## Four mistakes that make crawler logs misleading #

- **Blocking by user-agent string alone.** It refuses the real crawler and anyone impersonating it alike, and the impersonators simply change the string.
- **Treating unverifiable as fake.** A crawler whose operator publishes no list cannot be proven either way; counting it as forged inflates the forged share.
- **A range list copied once.** Ranges change; a list from last year will mark new, genuine addresses as spoofed.
- **Ignoring IPv6.** If your server logs IPv6 addresses, check them against the IPv6 entries, not only the IPv4 ones.

## What to do with each outcome #

| Outcome | Meaning | Action |
| --- | --- | --- |
| Verified and served | The real crawler read the page | None |
| Verified and refused (403, 429, challenge) | Your own rule is blocking the real crawler | Find the rule in the firewall, [CDN](/glossary/cdn) or security plugin and allow the published ranges |
| Spoofed | Someone else using the crawler’s name | Refusing it is correct; do not loosen the rule |
| Unverifiable | The operator publishes no list | Decide by behaviour (request rate, paths) rather than by name |

## Where CrawlCheck is weaker #

CrawlCheck checks the log lines you paste; it does not sit in front of your traffic, so it cannot block or rate-limit anything in real time. Cloudflare verifies every request at the edge as it arrives and can act on it, including through Web Bot Auth signatures. For sites on Cloudflare, use both: Cloudflare to enforce, CrawlCheck to audit logs from any host. Background: [how to verify Googlebot, Bingbot and AI crawlers by IP address](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers).
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Verify%20GPTBot%2C%20ClaudeBot%20and%20PerplexityBot%3A%20tools%20compared)[Email](mailto:?subject=Verify%20GPTBot%2C%20ClaudeBot%20and%20PerplexityBot%3A%20tools%20compared&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Related findings

[2026-11-13 Claude-User uses two published address lists. We checked one CrawlCheck checked Claude-User against one of two Anthropic address lists and counted real fetches as failed. Fixed 13 November 20](/blog/claude-user-comes-from-two-published-address-lists-we-checked-one)[2026-08-15 How much AI crawler traffic is forged? We check the IPs Requests claiming GPTBot, ClaudeBot or Googlebot, checked against each operator's published IP ranges: the live forged rate and ho](/blog/how-much-ai-crawler-traffic-is-forged)[2026-10-03 How to test robots.txt for GPTBot and ClaudeBot: 5 tools compared How to test robots.txt for GPTBot, ClaudeBot and PerplexityBot: five tools compared, a two-minute test, and why delivery matters m](/guides/how-to-test-robots-txt-for-gptbot-and-claudebot)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [/](/) · [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) · [TLS posture](/data/tls-posture) | **Blog:** [AI picked a local contractor 422 times: rating decided it](/blog/ai-picked-a-local-contractor-422-times-the-rating-decided-it) | **AI crawlers:** [Cloudflare AI search external](/ai-crawlers/cloudflare-ai-search-external) | **Glossary:** [AI classifiers: Scoring](/glossary/ai-classifiers/scoring) · [Document ingestion](/glossary/document-ingestion) · [Connection timeout](/glossary/connection-timeout) · [Redirected internal link](/glossary/redirected-internal-link) · [preflight](/glossary/preflight)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/how-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.