Agent view · Guides · 2026-10-02
How to verify Googlebot, GPTBot and other crawlers by IP
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 20,345 |
| words | 2,871 |
| sha-256 | e6343dcc8e3e89e68b5df834f1a414db3a253dcb4bedd1b2be62028fbf7e0a20 |
| repr-digest | sha-256=:5jQ9zI4+ieaLXfg08aQU2zolPctL7dGyvmICj79+CiA=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311500;expires=1795315100;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="BgfcipkrIr2aX-nYOLGqJQ";tag="crawlcheck-page" |
| signature | sig1=:mTlyVZ1Ml41pb3es/G59rs7T3+W3FID3MoYS0SXUAbrWPrzueB1CO2ACrejbmijVZRQasxSOB5EGZMe8hG05BA==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Guides · 2026-10-02 · By [VSNARY | Emmanuel Orta](/author) · 7 views # How to verify Googlebot, GPTBot and other crawlers by IP A user-agent string is a claim anyone can type. Verification is reverse DNS with a forward confirm, or a match against the operator's published IP list, and the answer has three outcomes, not two. To verify Googlebot, reverse-DNS the request's IP, confirm the name is under googlebot.com (never googleusercontent.com, which any Google Cloud customer gets), then forward-resolve that name and confirm it returns the same IP; or match the IP against Google's common-crawlers.json. Bingbot works the same way with search.msn.com or bingbot.json. OpenAI, Anthropic, Perplexity, Apple and Amazon publish IP lists. Every check ends in one of three outcomes: verified, contradicted (the address is outside the operator's list), or unverifiable (the operator publishes nothing, or your copy of the list is missing or stale). Check your own site free Check my site About a minute. First scan free with your email, nothing installed. On this page - How do I verify a crawler with reverse DNS? - Which crawlers publish an IP list, and where? - Checking GPTBot or ClaudeBot lines in your own log - Verified, contradicted, unverifiable: what each outcome means - How much named crawler traffic is forged? - When a crawler cannot be verified - Doing it at scale - Behind a CDN - Signed requests are the next method - What to do with the result Every crawler announces itself with a user-agent string, and every user-agent string can be copied. On one site we measured, every GPTBot 403 in a week came from a single address that also called itself ClaudeBot, PerplexityBot and Googlebot in the same hour. Before you allow, block, rate-limit or count a crawler by name, check the request against something its operator controls: DNS, or a published IP list. This guide covers both, lists what each major crawler publishes, and shows how to read the result. ## How do I verify a crawler with reverse DNS? # Take the IP address from the log line and look up its PTR record. For Googlebot the name is `crawl-*.googlebot.com` or `geo-crawl-*.geo.googlebot.com`; Google's special-case crawlers use `rate-limited-proxy-*.google.com`. A name under `googleusercontent.com` is a Google Cloud customer's machine, not Google, even though Google's page lists the domain (see [our measurement](/blog/googles-googlebot-verification-rule-passes-any-google-cloud-server)). Then look up that hostname's A or AAAA record and confirm it returns the original IP. The forward step is the one that matters: anyone who controls an IP block can set its PTR record to say `googlebot.com`, but only Google can make `googlebot.com` names resolve to that IP. ``` $ host 66.249.66.1 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com. $ host crawl-66-249-66-1.googlebot.com crawl-66-249-66-1.googlebot.com has address 66.249.66.1 ``` Bing uses the same scheme with names ending in `search.msn.com`. DNS verification is exact but slow at scale, because each new IP costs two lookups, so cache the result per IP for a day or more. ## Which crawlers publish an IP list, and where? # Most large operators publish their crawler addresses as JSON lists of CIDR ranges. Matching an IP against a cached copy of the right list is fast and needs no DNS. The word that trips people up is right: Google publishes separate files for its common crawlers, special-case crawlers and user-triggered fetchers, and they are not interchangeable. `user-triggered-fetchers.json` covers fetches made from customers' Google Cloud projects, so an IP in that file proves the request came from Google's cloud, not that Google sent it. | Crawler | Operator publishes | Where | | --- | --- | --- | | Googlebot and common Google crawlers | DNS convention and IP list | `googlebot.com` / `common-crawlers.json` | | Google user-triggered agents | IP list | `user-triggered-agents.json`, `user-triggered-fetchers-google.json` | | Bingbot | DNS convention and IP list | `search.msn.com` / `bing.com/toolbox/bingbot.json` | | GPTBot, OAI-SearchBot, ChatGPT-User | IP lists, one per agent | `openai.com/gptbot.json`, `searchbot.json`, `chatgpt-user.json` | | ClaudeBot and Anthropic agents | IP list | `claude.com/crawling/bots.json` | | PerplexityBot, Perplexity-User | IP lists | `perplexity.com/perplexitybot.json`, `perplexity-user.json` | | Applebot | IP list | `search.developer.apple.com/applebot.json` | | Amazonbot | IP list | see [how to verify Amazonbot](/guides/how-to-verify-amazonbot-and-amzn-searchbot) | | Meta-ExternalAgent, [Bytespider](/glossary/bytespider), [CCBot](/glossary/ccbot) | Nothing to check against | [unverifiable](/glossary/unverifiable) | Refresh the lists daily. Operators add ranges without notice, and a stale list turns real crawler traffic into apparent forgeries. ## Checking GPTBot or ClaudeBot lines in your own log # Pull the crawler lines first. On most hosts `grep -iE 'gptbot|claudebot|perplexitybot|googlebot|bingbot|applebot|oai-searchbot' access.log` is enough; keep the raw lines, because you need both the source address and the user-agent from each. Then group the lines by source address before you read anything into them. A wall of GPTBot 403s reads like your host is blocking OpenAI until the grouping shows one address wearing seven crawler names. On that site, a 57% “block rate” turned out to be one forged address and zero blocked crawlers (the full case is in [group the 403s by IP first](/blog/we-almost-published-that-our-host-was-blocking-claude)). Then check each address against the list for the name it claims. Two real lines show both results: a request claiming GPTBot from `136.66.226.148` is outside every range OpenAI publishes, so someone else is wearing the name; a request claiming ClaudeBot from `216.73.216.24` is inside Anthropic's list and is what it says. CrawlCheck's free log verifier does this for pasted lines and does not store them. ## Verified, contradicted, unverifiable: what each outcome means # “Not verified” is not one state. A contradiction and a shrug are different inputs to a block decision, and a rule that treats them alike will refuse clients it has no evidence about. | Outcome | What you know | Typical cases | | --- | --- | --- | | Verified | The address is in the list or DNS name for the identity claimed | Right range; last address in a range; correct IPv6; an IPv4-mapped IPv6 address of a listed IPv4 | | Contradicted | The evidence conflicts with the claim | First address past a range boundary; an address in the same operator's list for a different agent; wrong IPv6 | | Unverifiable | There is nothing current to check against | Operator publishes nothing; your copy of the list failed to download or is stale; an unknown name | Two cases in that table deserve emphasis. A request claiming GPTBot from an address in OpenAI's ChatGPT-User list is an OpenAI request, but it is not GPTBot: match each name to its own list, or you hide the difference between a training crawler and a [user-triggered fetch](/glossary/user-triggered-fetch). And when your list download fails or goes stale, the honest result is unverifiable, not forged. Calling it forged would accuse real traffic on the strength of your own outage. ## How much named crawler traffic is forged? # The rate varies sharply by name, because forgers copy the names sites are most likely to allow. The figure below is recomputed from live telemetry on this site, with unverifiable claims kept out of the denominator. The per-crawler breakdown is in [how much AI crawler traffic is forged](/blog/how-much-ai-crawler-traffic-is-forged). **Citing this figure.** The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: `https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged#the-rate`. Sentence form, as of 2026-11-22: *11.5% of requests claiming a verifiable AI-crawler identity, measured by CrawlCheck on its own traffic (25509 of 221127 adjudicable claims), came from outside the operator's published IP ranges; the most-forged name was [Google-Extended](/glossary/google-extended) at 100%.* Exclude your own testing before you count. Early on, 78 of 85 “GPTBot” hits in our own records were our own test requests. A dataset that trusts the user-agent string is a dataset about who lies well. ## When a crawler cannot be verified # Some operators publish no ranges and no DNS convention. Bytespider and Meta-ExternalAgent are the most visible. A request carrying such a name is not verifiable either way: counting it as genuine inflates the crawler's apparent reach, and counting it as forged accuses real traffic. Keep those requests in their own bucket and out of every verified and forged rate, as described in [which crawlers cannot be verified](/blog/the-crawlers-nobody-can-check). ## Doing it at scale # Verify in two tiers. Keep a local copy of every operator's IP list, refreshed daily, and match each request's IP against the list for the name it claims; this is a CIDR lookup and costs microseconds. Only when the name has no list, or the IP is not in it, fall back to DNS, and cache the DNS result per IP for at least a day. Watch IPv6. Several operators crawl over IPv6, and a matcher that only parses IPv4 ranges will mark every IPv6 request unverified; normalise IPv4-mapped IPv6 addresses before matching. Keep the raw evidence: the IP, the claimed name, which list or DNS name verified it, and when the list was last refreshed. A verification result you cannot re-derive is one you cannot defend when someone disputes a block. If you build your own verifier, write the expected outcome for each test case down before you run it, and keep synthetic test cases separate from real captures. ## Behind a CDN # If the site sits behind a [CDN](/glossary/cdn) or reverse proxy, the IP in the origin's logs is the proxy's, not the crawler's. Verification has to run on the client IP the proxy forwards, such as `CF-Connecting-IP` on Cloudflare, and only when the request actually came from the proxy; otherwise anyone can send that header directly to the origin and claim any address. A CDN's own verified-bot classification is a reasonable first filter but not a substitute for your own records. ## Signed requests are the next method # IP verification breaks down for agents that run on shared cloud infrastructure, where the same addresses serve many tenants. The emerging fix is cryptographic: the agent signs each request with a key it publishes. That scheme is covered in [how signed crawler requests work](/guides/web-bot-auth-how-signed-crawler-requests-work). Until it is widespread, DNS and IP lists remain the checks most crawlers support. ## What to do with the result # If the contradicted lines are the ones being refused, your edge is doing its job. If verified lines are being refused, that is a real block on a real crawler: check bot-management rules, [WAF](/glossary/waf) rules and rate limits for the ranges involved, then re-test by [scanning the site](/). Allow-list verified crawlers at the firewall by IP list rather than by user-agent, so a forged name gets the treatment an unknown client gets, and rate-limit unverifiable names like any other client. When you report crawler traffic, report verified, contradicted and unverifiable separately. The [AI crawler list](/ai-crawlers) shows which method applies to each crawler. CrawlCheck’s dataset when this page loaded: 582,334 sites measured. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-verify-googlebot-bingbot-and-ai-crawlers%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-verify-googlebot-bingbot-and-ai-crawlers%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=How%20to%20verify%20Googlebot%2C%20GPTBot%20and%20other%20crawlers%20by%20IP)[Email](mailto:?subject=How%20to%20verify%20Googlebot%2C%20GPTBot%20and%20other%20crawlers%20by%20IP&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-verify-googlebot-bingbot-and-ai-crawlers)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### How do I verify that a request is really from Googlebot? Reverse-DNS the IP, confirm the name is crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com (not googleusercontent.com, which any Google Cloud customer gets), then forward-resolve that name and confirm it returns the same IP. Or match the IP against Google's common-crawlers.json. ### Why is the forward DNS lookup necessary? Anyone who controls an IP block can set its reverse DNS to claim googlebot.com. Only Google can make googlebot.com names resolve to an IP, so the forward lookup is the proof. ### How do I check whether a GPTBot or ClaudeBot request is real? Take the source address from the log line and check it against the list for that exact agent: openai.com/gptbot.json for GPTBot, claude.com/crawling/bots.json for ClaudeBot. Inside the list means verified; outside it means someone else is using the name. ### Is an IP in user-triggered-fetchers.json proof that Google sent the request? No. That file covers fetches from customers' Google Cloud projects, which anyone can run. It proves Google Cloud, not Google. ### What is the difference between contradicted and unverifiable? Contradicted means the operator publishes a list and the address is outside it. Unverifiable means there is nothing current to check against, because the operator publishes nothing or your copy of the list is missing or stale. Only the first is evidence of forgery. ### What if a crawler publishes no IP ranges? Then a request carrying its name cannot be verified. Classify it as unverifiable and keep it out of verified and forged counts alike. ## Related findings [2026-11-20 How to tell a real AI crawler from one borrowing its name Any request can call itself an AI crawler. Here is how to check the name against the source: reverse DNS, published address lists](/guides/how-to-tell-a-real-ai-crawler-from-one-borrowing-its-name)[2026-08-15 How much AI crawler traffic is forged? We check the IPs Requests claiming GPTBot, ClaudeBot or Googlebot, checked against each operator's published IP ranges: the live forged rate and ho](/blog/how-much-ai-crawler-traffic-is-forged)[2026-10-03 Verify GPTBot, ClaudeBot and PerplexityBot: tools compared How to verify GPTBot, ClaudeBot and PerplexityBot requests: published IP lists, CrawlCheck's free log check and Cloudflare verifie](/guides/how-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Registry:** [AI policy](/registry/aipolicy) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) · [Conduct: OAI-SearchBot](/agents/conduct/oai-searchbot) | **Datasets:** [Technologies: Plausible](/datasets/technologies/plausible) · [Technologies: Google maps](/datasets/technologies/google-maps) · [Technologies: Livechat](/datasets/technologies/livechat) | **Blog:** [Four AI models, three local businesses: who got it right](/blog/what-four-ai-models-told-customers-about-our-three-businesses) | **Glossary:** [TTFB](/glossary/ttfb) · [Dot-product similarity](/glossary/dot-product-similarity) · [If-None-Match](/glossary/if-none-match) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /guides/how-to-verify-googlebot-bingbot-and-ai-crawlers.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.