Guides · 2026-09-02 · By VSNARY | Emmanuel Orta
How to check whether a GPTBot or ClaudeBot request in your logs is real
A user-agent is a claim. The address is the evidence. Here is how to adjudicate every crawler line in an access log in one paste, and how to read the three outcomes.
Anyone can send a request with GPTBot in the user-agent string. Scrapers do it to inherit whatever allowances a site grants OpenAI; vulnerability scanners do it to look routine. A log full of GPTBot 403s therefore reads like your host is blocking OpenAI — until you group the lines by source address and find every one came from a single IP that also called itself ClaudeBot, PerplexityBot and Googlebot the same hour. We measured exactly that on a live site. The name is a claim; the address is the evidence.
What makes a claim checkable
Several operators publish the IP ranges their crawlers use, as machine-readable JSON: OpenAI for GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic for ClaudeBot and its agents; Perplexity; Google; Microsoft for Bingbot; Apple for Applebot. A request that names one of those crawlers is verified if its source address falls inside the operator’s published ranges and spoofed if it falls outside all of them. Some operators publish nothing — Amazonbot and Meta-ExternalAgent among them — so a request naming those can be neither confirmed nor accused. The crawler reference lists which is which.
Step 1: pull the lines
From your access log, take the lines that mention a crawler name. On most hosts something like grep -iE 'gptbot|claudebot|perplexitybot|googlebot|bingbot|applebot|oai-searchbot' access.log is enough. Keep the raw lines; the verifier needs the source address and the user-agent from each, and it reads both from standard combined-log format.
Step 2: paste them into the verifier
Open the free tools page and paste the lines into the log verifier, one per line, up to 200. Nothing you paste is stored; the lines are adjudicated in memory and the response is not logged. Or call the endpoint directly:
curl -s -X POST https://crawlcheck.io/api/tool/verify \
-H 'content-type: application/json' \
-d '{"lines": "<paste lines here, newline separated>"}'The lines field accepts a newline-separated string or a JSON array of strings. The response carries a verdict per line and a tally.
Step 3: read the three outcomes
Verified: the address is inside the range the operator publishes. The request is what it says it is. Spoofed: the operator publishes ranges and this address is outside all of them; someone else is wearing the name. Unverifiable: the operator publishes no feed, so the claim cannot be settled either way. The forgery rate the response reports divides by verified plus spoofed only. Counting the unverifiable as forged would inflate it; counting them as genuine would understate it; so they are excluded, and the response says so in its denominator note.
A worked example from real lines: a request claiming GPTBot from 136.66.226.148 comes back spoofed, because that address is outside every range OpenAI publishes; a request claiming ClaudeBot from 216.73.216.24 comes back verified. The same tool, run on a week of one site’s logs, turned a 57% “block rate” into one forged address and zero blocked crawlers.
What to do with the answer
If the spoofed lines are the ones being refused, your edge is doing its job and there is nothing to fix. If verified lines are being refused, that is a real block on a real crawler: check bot-management rules, WAF rules and rate limits for the address ranges involved, and re-test by scanning the site, which fetches the page as each named crawler and reports what the edge did by name. If the picture is mostly unverifiable, do not conclude anything from it; group by address and look at what else each address requested.
Why the scanner never counts this for you
CrawlCheck’s own telemetry applies the same rule to its own site: a hit is counted as a crawler only when the address verifies, and our own verification probes are marked so they are never filed as GPTBot visits. Early in the project 78 of 85 recorded “GPTBot” hits turned out to be our own curls. A dataset that trusts the user-agent string is a dataset about who lies well, which is why every crawler figure published here states its denominator.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Questions this post answers
Which crawlers can be verified by IP?
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and Anthropic’s agents, PerplexityBot, Googlebot and Google’s fetchers, Bingbot and Applebot all have published ranges. Amazonbot, Meta-ExternalAgent, Bytespider and CCBot do not, so requests naming them are unverifiable.
Is the verifier free and does it keep my logs?
Free, no account, and nothing is stored: up to 200 lines are adjudicated in memory and the response is not logged.
Why is the forgery rate not spoofed divided by all lines?
Because unverifiable lines cannot be called either way. The rate divides by verified plus spoofed only, and the response states that denominator so the number can be reproduced.
A verified crawler is being refused. Is that cloaking?
No. Refusing a request is a policy or a misconfiguration, not cloaking. Cloaking is serving a crawler different content from what a visitor gets; the scanner tests for that separately by fetching the page as each crawler and as a browser in the same second.