Guides · 2026-08-28 · By VSNARY | Emmanuel Orta

The AI crawler list: what each one is for, and which can be verified

Twelve names that show up in server logs, what each is actually doing, and the number that changes how you read all of them — how much of that traffic is not who it says it is.

Three separate jobs get lumped together under AI crawler, and confusing them leads to blocking the wrong one.

The third category is the one most often blocked by accident. A user-triggered fetch is not a scraper harvesting your site — it is a person asking an assistant about you, and the assistant going to look. Refusing it means the answer gets composed without you.

The names, and whether the claim is checkable

Verification matters more than the list itself. OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish IP ranges, so a request claiming those identities can be checked. Others publish nothing, which does not mean the traffic is fake — it means the claim cannot be tested either way.

Across 2,510 checkable claims, 928 — 37% — came from an address outside the range the operator publishes. Four identities in the wider dataset came back at 100% forged: Google-Extended, Claude-SearchBot, anthropic-ai and Applebot-Extended. Those are the retired or training-specific names, which makes sense: nobody is running them any more, so everything wearing the name is something else.

Unverifiable is not forgery. The five rows with no feed are not accused of anything; there is simply no published range to test them against, so a claim in their name is untestable rather than false.

What the forgery number changes

Three practical consequences, and they are the whole reason this list is worth having.

Your log volume is not a health signal. If nearly half the traffic calling itself GPTBot is not GPTBot, then a busy-looking crawl log tells you very little about whether you are being read.

Never write an allow rule that trusts a user-agent. A rule anyone can satisfy by typing a string is not a rule. Allow by verified range, or by nothing.

Some of it is hostile. Forged crawler identities in this dataset have been observed requesting .env files, SSH keys and other credential paths. A crawler name is a common disguise precisely because site owners have been taught to let those requests through.

The identities that came back 100% forged

Four names in the wider dataset returned a forgery rate of exactly 100%: Google-Extended, Claude-SearchBot, anthropic-ai and Applebot-Extended. Every checkable request wearing those names came from outside the operator's published range.

That result is less surprising than it reads. Those are the training-specific and superseded identities — names that were widely publicised, widely added to block lists, and are now little used by the operators themselves. What remains in the wild is everything that picked the name up second-hand: scanners, scrapers and tooling that copied a user-agent string out of a blog post.

The practical read is that the more famous a crawler name became, the less its appearance in your logs tells you. One of those identities was measured requesting only credential paths — .env files, SSH keys, config — with just one request in ten going to an ordinary page. That is not a crawler with an unusual interest in your site. It is a scanner wearing a name it expects you to have whitelisted.

Verified status is not permanent either

Two developments are worth holding alongside the table, because both change what a name in your logs means.

First, an operator can lose verified status. Cloudflare removed Perplexity from its verified bot list after investigating reports of crawling that continued past configured blocks; Perplexity disputed the account. Whatever the merits, the structural lesson stands: the verified-bot lists that infrastructure providers maintain are editorial, they change, and a policy written against a name inherits whatever that list decides later.

Second, the categories are being redrawn around purpose rather than identity. From 15 September 2026, Cloudflare's defaults split crawlers into Search, Agent and Training, and blocking Training also blocks multi-purpose crawlers — Googlebot, Applebot, Bingbot — that collect for both search and training. A block aimed at one job now reaches names you may have assumed were on a different list entirely.

Together those make identity-based policy weaker than it looks. A rule that names a bot is a rule about a label that a third party controls, an operator can retire, and a forger can wear.

Working out which ones actually visit you

General lists are a starting point, not an answer. Which crawlers arrive at a given site varies enormously with what the site is and who links to it, and the only way to know yours is to read your own access logs — verifying each request's source address against the operator's published range before you count it.

Do that before you make policy. A block list built from a general article will restrict crawlers that never visit you, and leave alone the ones that do.

Questions this post answers

What are the main AI crawlers?

The frequently seen verifiable ones are GPTBot and OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot from Anthropic, PerplexityBot from Perplexity, Applebot from Apple, and Googlebot. Meta-ExternalAgent, Amazonbot, AhrefsBot, SemrushBot and DataForSeoBot also appear but publish no IP range to verify against.

Which AI crawlers can be verified?

OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish IP ranges, so a request claiming those identities can be checked against the source address. Operators without a published feed cannot be verified either way, which is not the same as the traffic being fake.

Should I block AI crawlers?

Separate the three jobs first. Blocking training crawlers has no immediate cost, but blocking search-indexing or user-triggered fetches removes your eligibility to be cited, including when a person is asking an assistant about you right now.

Related findings

All guides · The dataset · How the dataset works