Guides · 2026-09-02 · By VSNARY | Emmanuel Orta · 1 view

How to check whether a GPTBot or ClaudeBot request in your logs is real

A user-agent is a claim. The address is the evidence. Here is how to adjudicate every crawler line in an access log in one paste, and how to read the three outcomes.

Across every full scan CrawlCheck has run, an answer engine's crawler was refused while a browser was served on 5.4% of sites, which is the case where your verified log lines, not the spoofed ones, show the refusals.

Anyone can send a request with GPTBot in the user-agent string. Scrapers do it to inherit whatever allowances a site grants OpenAI; vulnerability scanners do it to look routine. A log full of GPTBot 403s therefore reads like your host is blocking OpenAI — until you group the lines by source address and find every one came from a single IP that also called itself ClaudeBot, PerplexityBot and Googlebot the same hour. We measured exactly that on a live site. The name is a claim; the address is the evidence.

What makes a claim checkable #

Several operators publish the IP ranges their crawlers use, as machine-readable JSON: OpenAI for GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic for ClaudeBot and its agents; Perplexity; Google; Microsoft for Bingbot; Apple for Applebot. A request that names one of those crawlers is verified if its source address falls inside the operator’s published ranges and spoofed if it falls outside all of them. Some operators publish nothing — Amazonbot and Meta-ExternalAgent among them — so a request naming those can be neither confirmed nor accused. The crawler reference lists which is which.

Step 1: pull the lines #

From your access log, take the lines that mention a crawler name. On most hosts something like grep -iE 'gptbot|claudebot|perplexitybot|googlebot|bingbot|applebot|oai-searchbot' access.log is enough. Keep the raw lines; the verifier needs the source address and the user-agent from each, and it reads both from standard combined-log format.

Step 2: paste them into the verifier #

Open the free tools page and paste the lines into the log verifier, one per line, up to 200. Nothing you paste is stored; the lines are adjudicated in memory and the response is not logged. Or call the endpoint directly:

curl -s -X POST https://crawlcheck.io/api/tool/verify \
  -H 'content-type: application/json' \
  -d '{"lines": "<paste lines here, newline separated>"}'

The lines field accepts a newline-separated string or a JSON array of strings. The response carries a verdict per line and a tally.

Step 3: read the three outcomes #

Verified: the address is inside the range the operator publishes. The request is what it says it is. Spoofed: the operator publishes ranges and this address is outside all of them; someone else is wearing the name. Unverifiable: the operator publishes no feed, so the claim cannot be settled either way. The forgery rate the response reports divides by verified plus spoofed only. Counting the unverifiable as forged would inflate it; counting them as genuine would understate it; so they are excluded, and the response says so in its denominator note.

A worked example from real lines: a request claiming GPTBot from 136.66.226.148 comes back spoofed, because that address is outside every range OpenAI publishes; a request claiming ClaudeBot from 216.73.216.24 comes back verified. The same tool, run on a week of one site’s logs, turned a 57% “block rate” into one forged address and zero blocked crawlers.

What to do with the answer #

If the spoofed lines are the ones being refused, your edge is doing its job and there is nothing to fix. If verified lines are being refused, that is a real block on a real crawler: check bot-management rules, WAF rules and rate limits for the address ranges involved, and re-test by scanning the site, which fetches the page as each named crawler and reports what the edge did by name. If the picture is mostly unverifiable, do not conclude anything from it; group by address and look at what else each address requested.

Before you run this on your own log, here is what the same procedure returns on ours. These are live figures from requests to this domain, adjudicated by the method described above rather than asserted.

Forged crawler identities #

6.6% of the requests that named themselves as a known crawler here were not that crawler. 3703 of 55824 checkable claims came from an address outside the range the operator publishes.

A user-agent is a claim, not an identity. Verified means the source IP sits inside a range the operator publishes — OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish one. Unverifiable is not forgery: some operators publish no range at all, so their requests can be neither confirmed nor accused, and they are excluded from the rate rather than counted against it. This is traffic to this site only, and this site is small — it is a floor on the problem, not a survey of the web.

Why CrawlCheck never counts this for you #

CrawlCheck’s own telemetry applies the same rule to its own site: a hit is counted as a crawler only when the address verifies, and our own verification probes are never filed as GPTBot visits. Early in the project 78 of 85 recorded “GPTBot” hits turned out to be our own curls. A dataset that trusts the user-agent string is a dataset about who lies well, which is why every crawler figure published here states its denominator.

Questions this post answers

Which crawlers can be verified by IP?

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and Anthropic’s agents, PerplexityBot, Googlebot and Google’s fetchers, Bingbot and Applebot all have published ranges. Amazonbot, Meta-ExternalAgent, Bytespider and CCBot do not, so requests naming them are unverifiable.

Is the verifier free and does it keep my logs?

Free, no account, and nothing is stored: up to 200 lines are adjudicated in memory and the response is not logged.

Why is the forgery rate not spoofed divided by all lines?

Because unverifiable lines cannot be called either way. The rate divides by verified plus spoofed only, and the response states that denominator so the number can be reproduced.

A verified crawler is being refused. Is that cloaking?

No. Refusing a request is a policy or a misconfiguration, not cloaking. Cloaking is serving a crawler different content from what a visitor gets; CrawlCheck tests for that separately by fetching the page as each crawler and as a browser.

Why does my access log show so many GPTBot 403 errors?

Often because the requests are not from OpenAI. Anyone can put GPTBot in a user-agent string, and scrapers and vulnerability scanners do it to borrow a crawler's allowances or look routine. Group the 403 lines by source address and check each against OpenAI's published ranges. If the refused lines are spoofed, your edge is working. If verified GPTBot addresses are refused, review your bot-management, WAF and rate-limit rules.

How do I pull crawler requests out of an access log?

Filter the log for crawler names and keep the raw lines. On most hosts a command such as grep -iE 'gptbot|claudebot|perplexitybot|googlebot|bingbot|applebot|oai-searchbot' access.log is enough. Keep each full line, because verification needs both the source IP and the user-agent, and standard combined-log format carries both. Then check each address against the operator's published IP ranges, by hand or with a log verifier.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works