Every agent that reads the web, and whether a request is really it.

The identity reference for the agentic web, read by sites deciding what to serve and by agents deciding what to trust: every crawler and fetcher this verification layer recognises — 115 policy identities resolved from robots.txt, of which 15 are represented by requests in the full scan — grouped by what each one is for, with how a request wearing the name is checked against the ranges its operator publishes, and how each agent has actually behaved where it was measured. The same company usually sends three different crawlers for three different purposes, and blocking one does not do what people expect the others to do.

Counted, never judged: Meta-ExternalAgent 67,933 · SemrushBot 22,178 · DataForSeoBot 13,651 · Bytespider 9,964 · YouBot 2,058 · cohere-ai 51 · and 7 more — their operators publish no address ranges, so none of these requests can be told from an impersonator.

This page is the reference. The measurement is separate: a scan sends 19 request profiles to your homepage — GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User, Googlebot, Googlebot Smartphone, Bingbot, Applebot, Amazonbot, Bytespider, Meta-ExternalAgent, CCBot, plus an unnamed client and two mobile-browser controls; the other named agents below are resolved from your robots.txt, not sent — and records what each one was actually served. A difference between two of them is your edge deciding what each one gets. It also resolves every named agent in your own robots.txt into what it is permitted to do, which is how a policy someone wrote two years ago turns out to say something they did not intend.

Operator, purpose, and whether that operator publishes the IP ranges that make a claimed identity checkable. A crawler with no published feed is unverifiable here — never counted as forged.

Six groups, and the grouping is the point: an agent that trains a model, one that builds a search index and one that fetches your page at the moment a person asks a question are three different things with three different consequences. The third column is the one to read — it says whether a request wearing that name can be checked at all.

A name reaches the tables above only with a first-party operator page, an observed request, or a verifiable directory record. These have been seen in community blocklists or product notes and fail that gate today. They are not resolved from robots.txt and not sent to any site; each graduates on its own evidence, never as a batch.

curl -sI -A "GPTBot/1.2" https://yoursite.com/robots.txt

Which AI crawlers actually visit a small business site →The file that returned 200 and could not be read →The dataset →