Guides · 2026-08-28 · By VSNARY | Emmanuel Orta · 0 views

The AI crawler and agent list: what each one is for, and which can be verified

Twelve names that show up in server logs, what each is actually doing, and the number that changes how you read all of them — how much of that traffic is not who it says it is.

Three separate jobs get lumped together under AI crawler, and confusing them leads to blocking the wrong one.

The third category is the one most often blocked by accident. A user-triggered fetch is not a scraper harvesting your site — it is a person asking an assistant about you, and the assistant going to look. Refusing it means the answer gets composed without you.

The names, and whether the claim is checkable #

Verification matters more than the list itself. OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish IP ranges, so a request claiming those identities can be checked. Others publish nothing, which does not mean the traffic is fake — it means the claim cannot be tested either way.

Across 2,510 checkable claims, 928 — 37% — came from an address outside the range the operator publishes. Four identities in the wider dataset came back at 100% forged: Google-Extended, Claude-SearchBot, anthropic-ai and Applebot-Extended. Those are the retired or training-specific names, which makes sense: nobody is running them any more, so everything wearing the name is something else.

Unverifiable is not forgery. The five rows with no feed are not accused of anything; there is simply no published range to test them against, so a claim in their name is untestable rather than false.

What the forgery number changes #

Three practical consequences, and they are the whole reason this list is worth having.

Your log volume is not a health signal. If nearly half the traffic calling itself GPTBot is not GPTBot, then a busy-looking crawl log tells you very little about whether you are being read.

Never write an allow rule that trusts a user-agent. A rule anyone can satisfy by typing a string is not a rule. Allow by verified range, or by nothing.

Some of it is hostile. Forged crawler identities in this dataset have been observed requesting .env files, SSH keys and other credential paths. A crawler name is a common disguise precisely because site owners have been taught to let those requests through.

The identities that came back 100% forged #

Four names in the wider dataset returned a forgery rate of exactly 100%: Google-Extended, Claude-SearchBot, anthropic-ai and Applebot-Extended. Every checkable request wearing those names came from outside the operator's published range.

That result is less surprising than it reads. Those are the training-specific and superseded identities — names that were widely publicised, widely added to block lists, and are now little used by the operators themselves. What remains in the wild is everything that picked the name up second-hand: scanners, scrapers and tooling that copied a user-agent string out of a blog post.

The practical read is that the more famous a crawler name became, the less its appearance in your logs tells you. One of those identities was measured requesting only credential paths — .env files, SSH keys, config — with just one request in ten going to an ordinary page. That is not a crawler with an unusual interest in your site. It is a scanner wearing a name it expects you to have whitelisted.

Verified status is not permanent either #

Two developments are worth holding alongside the table, because both change what a name in your logs means.

First, an operator can lose verified status. Cloudflare removed Perplexity from its verified bot list after investigating reports of crawling that continued past configured blocks; Perplexity disputed the account. Whatever the merits, the structural lesson stands: the verified-bot lists that infrastructure providers maintain are editorial, they change, and a policy written against a name inherits whatever that list decides later.

Second, the categories are being redrawn around purpose rather than identity. From 15 September 2026, Cloudflare's defaults split crawlers into Search, Agent and Training, and blocking Training also blocks multi-purpose crawlers — Googlebot, Applebot, Bingbot — that collect for both search and training. A block aimed at one job now reaches names you may have assumed were on a different list entirely.

Together those make identity-based policy weaker than it looks. A rule that names a bot is a rule about a label that a third party controls, an operator can retire, and a forger can wear.

Working out which ones actually visit you #

General lists are a starting point, not an answer. Which crawlers arrive at a given site varies enormously with what the site is and who links to it, and the only way to know yours is to read your own access logs — verifying each request's source address against the operator's published range before you count it.

Do that before you make policy. A block list built from a general article will restrict crawlers that never visit you, and leave alone the ones that do.

Twelve names is the short list. The policy list is 114. #

The table above covers what a small site actually sees. A robots.txt file, a firewall rule or an allowlist has to cover more than that, because policy applies to every identity that might arrive, not only the ones that already have. The catalogue behind this guide holds 114 named crawler identities, each taken from its operator’s own documentation.

A name enters that catalogue on one condition: the operator’s own page names the token. Log volume is not the entry test. If it were, the list would be a ranking of whoever was loudest last month — and the forgery rates in the table above are exactly why that would be the wrong list.

Seven jobs, not three #

The three-way split at the top of this guide is the version worth remembering. The full breakdown is finer, and every identity carries a job derived from what its operator says it does:

Control tokens are the two entries that confuse rule writers most. They are names you put in robots.txt to express a preference; nothing ever arrives calling itself one. A rule that blocks a control token blocks no traffic, and a log line that appears to contain one is telling you about the sender, not about the operator.

The verification column decides what a log can tell you #

Every identity also carries how it can be checked, derived from whether its operator publishes anything to check against:

Eighty-three of 114 — more than seven in ten — have no published way to be authenticated at all. That is the practical ceiling on identity-based policy, and it is worth saying plainly: an allow rule keyed on one of those names is an allow rule keyed on a string that anybody can type.

The same list, machine-readable #

If you are generating rules rather than reading a table, take /ai-crawlers.json instead. It carries the same list this page renders, licensed CC BY 4.0, and served with open CORS so a browser tool can read it directly. Each entry carries the token, operator, description, purpose, job type, verification method, verification source, and a boolean for whether it can be verified at all. The response also carries the totals and the schema itself, so a consumer can notice a new job type rather than silently dropping it.

What keeps a name off the list #

Ten further candidates sit on a published watchlist with their likely operator, their likely role, and the specific condition that would admit them. They are deliberately not counted in the 114.

Two recent examples of why. One token’s own documentation URL returned 410 Gone when we went to check it. Another operator’s site refused our request outright. Both are probably real crawlers. Neither is admissible, because the gate is the operator’s own page resolving and saying so — not a sighting, not a screenshot, not a mention in somebody else’s list.

The cost of that rule is that the catalogue lags reality slightly. The benefit is that nothing in it is there because somebody saw it once in a log file, which is more or less how the names in the 100%-forged group earned their reputations.

Verifiable, verified and welcome are three different words #

The table near the top of this guide splits identities into verifiable and not. That is the first of three states, and collapsing them is where most log analysis goes wrong.

That third row is the one people quietly convert into the second. A request from an operator with no published feed is not a forgery; it is a claim nobody can adjudicate, and folding it into a forgery percentage inflates the number with cases that were never testable. Every rate quoted in this guide is computed over adjudicable claims only, which is why the unverifiable identities show a dash rather than a zero.

There is a second guard on the same principle: an identity is never named in a rate until there are at least ten adjudicable claims behind it. Without that floor, a crawler seen twice and forged twice reports as 100% forged, which is arithmetically true and useless. Two of the three biggest surprises in our own early data were that shape.

And none of this speaks to whether you want the traffic. Verified means the request is really from who it says. Whether that operator should be reading your site is a policy question, and it is the one place on this page where the answer is genuinely yours rather than measurable.

Crawlers, fetchers and agents are three different entries

The list above is organised by operator. The more useful split for a site owner is by who decides that a page gets fetched, because that decides which control applies.

Crawlers read; agents act. A crawler deciding what to index and an agent filling in your quote form for someone are both “AI traffic”, and only the first is governed well by robots.txt.

Findings behind this #

Every figure above comes from a measurement written up in full, with the date, the method and the raw numbers.

• How much AI crawler traffic is forged? Checked against IP ranges
• Which AI crawlers publish IP ranges, and which cannot be verified
• Fake GPTBot and ClaudeBot requests are scanning for .env and .ssh files
• Which AI crawlers actually visit a small business site

Questions this post answers

What are the main AI crawlers?

The frequently seen verifiable ones are GPTBot and OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot from Anthropic, PerplexityBot from Perplexity, Applebot from Apple, and Googlebot. Meta-ExternalAgent, Amazonbot, AhrefsBot, SemrushBot and DataForSeoBot also appear but publish no IP range to verify against.

Which AI crawlers can be verified?

OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish IP ranges, so a request claiming those identities can be checked against the source address. Operators without a published feed cannot be verified either way, which is not the same as the traffic being fake.

Should I block AI crawlers?

Separate the three jobs first. Blocking training crawlers has no immediate cost, but blocking search-indexing or user-triggered fetches removes your eligibility to be cited, including when a person is asking an assistant about you right now.

How many AI crawlers are there?

There are 114 named crawler identities on record here, each taken from its operator’s own documentation, and they do seven different jobs: training, search indexing, answer-engine retrieval, tool and developer fetches, link previews, user-triggered fetches, and control tokens that nothing ever sends. The count moves as operators publish new tokens, and the number shown on the page is read from the list itself rather than typed into the text.

Can I verify an AI crawler from my server logs?

Only for the minority whose operator publishes something to check against. Twenty-nine of the 114 identities carry a published IP feed, a reverse-DNS convention, a documented ASN range or equivalent operator documentation. For the remaining 83 the user-agent string is the only evidence there is, and anyone can send a user-agent string.

Is there a machine-readable list of AI crawler user agents?

Yes. /ai-crawlers.json is generated from the same list this page renders, licensed CC BY 4.0 and served with open CORS. Each entry carries the token, the operator, the job type, how it can be verified and whether it can be verified at all, so a rule generator can read it directly instead of scraping a table.

What is the difference between an AI crawler and an AI agent?

A crawler fetches pages on its operator's schedule to train or index. An agent fetches because a person asked it to do something now: read a page, compare prices, fill in a form. User-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User are the agent-side clients a site sees most often.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works