CrawlCheck

Agent view · Findings · 2026-11-13

AI crawler IP ranges from 18 operator lists, in one file

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/ai-crawler-ip-ranges-from-18-operator-lists-in-one-file: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/ai-crawler-ip-ranges-from-18-operator-lists-in-one-file>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes11,165
words1,506
sha-256d179274714a2899dc91d0e9defe6ae2f537679c8581a4b12c45b52890d395f4d
repr-digestsha-256=:0XknRxSiiZ3JHQ6d7+auL1N2echYGksSxFtSiQ05X00=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795180198;expires=1795183798;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="Phj-K89RuL4IhXHLvUts8g";tag="crawlcheck-page"
signaturesig1=:qf7teM3GdSjZHObwyiyKWBggf5f4uQ7DY1VnN6iS5nMLfMwtLHIFK7Hkfj0Ji8KUXVAko5pibY6VgCnRxOwZCQ==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Findings · 2026-11-13 · By [VSNARY | Emmanuel Orta](/author)

# AI crawler IP ranges from 18 operator lists, in one file

Checking whether a crawler is real means comparing its address with the operator's list. Those lists live at 18 different addresses in several formats. We now mirror them in one file.

CrawlCheck now publishes one JSON file that mirrors 18 address lists published by crawler operators: OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Mistral and Common Crawl. It is refreshed hourly. Each source shows whether the last read worked, its HTTP status and a SHA-256 of what was read. On 13 November 2026 it held 3,963 IPv4 and 408 IPv6 prefixes from 18 of 18 sources.

On this page

- What is in it
- How to read a source
- What it is not
- Using it

Address lists mirrored, per operator
OpenAI 4

Google 3

Amazon 3

Perplexity 2

Mistral 2

Anthropic 1

Microsoft 1

Apple 1

Common Crawl 1

A [user agent](/glossary/user-agent) is a claim. Anyone can send a request that says GPTBot. The usual way to check the claim is to compare the request's address with the list the operator publishes. The trouble is the lists: each operator keeps its own, at its own address, in its own shape, and two of Amazon's are web pages of single addresses.

The file is at [crawlcheck.io/ai-crawlers/ip-ranges.json](/ai-crawlers/ip-ranges.json). It is free and needs no key.

## What is in it #

| Operator | Lists mirrored | Crawlers covered |
| --- | --- | --- |
| OpenAI | 4 | GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot |
| Anthropic | 1 | ClaudeBot, Claude-SearchBot, Claude-User |
| Perplexity | 2 | PerplexityBot, Perplexity-User |
| Google | 3 | Googlebot and its common crawlers, [Google-Agent](/glossary/google-agent), Google-GeminiNotebook |
| Amazon | 3 | Amazonbot, Amzn-SearchBot, Amzn-User |
| Mistral | 2 | MistralAI-User, MistralAI-Index |
| Microsoft | 1 | bingbot |
| Apple | 1 | Applebot |
| Common Crawl | 1 | [CCBot](/glossary/ccbot) |

## How to read a source #

Every source carries the address we read, the crawlers it covers, whether the read worked, the HTTP status and a [SHA-256](/glossary/sha-256) of the response. If an operator's server fails, the source is marked as failed and keeps its last good list with the time that list was read, flagged as stale. You can tell a fresh list from an old one without guessing.

## What it is not #

It is a mirror, not an authority. The operator's own page is the source, and each entry links to it. The file is cached for an hour, so a change on an operator's side can take up to an hour to appear.

It also holds only what operators publish as crawler lists. Anthropic's on-demand fetcher can arrive from a second published list that is not a crawler list, which is how [we got it wrong ourselves](/blog/claude-user-comes-from-two-published-address-lists-we-checked-one).

## Using it #

Fetch the file on a schedule, load the prefixes for the crawlers you care about, and compare each request's address before you trust its user agent. If you block at the server, pair the address check with [a user agent rule](/guides/block-ai-crawlers-nginx-apache-caddy): one stops the named crawler, the other stops things pretending to be it.

CrawlCheck’s dataset when this page loaded: 570,959 sites measured.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fai-crawler-ip-ranges-from-18-operator-lists-in-one-file%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fai-crawler-ip-ranges-from-18-operator-lists-in-one-file%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=AI%20crawler%20IP%20ranges%20from%2018%20operator%20lists%2C%20in%20one%20file)[Email](mailto:?subject=AI%20crawler%20IP%20ranges%20from%2018%20operator%20lists%2C%20in%20one%20file&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fai-crawler-ip-ranges-from-18-operator-lists-in-one-file)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Where can I get AI crawler IP ranges in one place?

At crawlcheck.io/ai-crawlers/ip-ranges.json. It mirrors 18 lists published by OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Mistral and Common Crawl.

### How often is the file updated?

It is re-read from the operators' own pages and cached for one hour.

### What happens when an operator's list is unreachable?

That source is marked as failed and keeps its last good list, flagged as stale, with the time it was last read successfully.

### Is an address check enough to verify a crawler?

It is the strongest check most sites can run. A request whose address is on the operator's list for that crawler is very likely real; one that is not is either forged or relayed through something else.

## Related findings

[2026-11-13 Release notes: 7 to 13 November 2026 CrawlCheck release notes for 7 to 13 November 2026: block rules for four servers, crawler IP ranges in one file, catalogue correct](/blog/release-notes-7-to-13-november-2026)[2026-11-13 Claude-User uses two published address lists. We checked one CrawlCheck checked Claude-User against one of two Anthropic address lists and counted real fetches as failed. Fixed 13 November 20](/blog/claude-user-comes-from-two-published-address-lists-we-checked-one)[2026-10-02 How to verify Googlebot, GPTBot and other crawlers by IP Verify Googlebot, Bingbot, GPTBot and ClaudeBot by reverse DNS or the operator's IP list. The file for each crawler, the traps, an](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [Cloudflare](/fix/cloudflare) · [Visitor resolve](/spec/visitor-resolve) | **Datasets:** [Problems: Declared sitemap broken](/datasets/problems/declared-sitemap-broken) | **Blog:** [A slider that took 8.8 seconds: three 328 KB PNGs](/blog/a-slider-that-took-8-seconds-three-328-kb-pngs) | **Glossary:** [agentic knowledge graph](/glossary/agentic-knowledge-graph) · [Sitemap URL set](/glossary/sitemap-url-set) · [Topic cluster](/glossary/topic-cluster) · [C2PA assertion](/glossary/c2pa-assertion)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /blog/ai-crawler-ip-ranges-from-18-operator-lists-in-one-file.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.