AI crawlers · Training crawlers

`anthropic-ai`

anthropic-ai is the training crawler run by Anthropic: Anthropic legacy agent. Its operator publishes no IP ranges or DNS names for it, so a request carrying this name cannot be verified. Treat the name in a log line as a claim.

User-agent token	`anthropic-ai`
Operator	Anthropic
What it does	Anthropic legacy agent
Kind	training crawler
Can a request be verified?	No — user agent string only
In a CrawlCheck scan	Read from robots.txt; reported, never scored

Allow or block it in robots.txt

User-agent: anthropic-ai
Disallow: /

User-agent: anthropic-ai
Allow: /

Add a group that names it: "User-agent: anthropic-ai" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names anthropic-ai, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.

What blocking it changes

No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.

What it actually did on crawlcheck.io

Verified requests	0
Requests using its name from outside its operator's ranges	4 (100% of claims)
robots.txt fetched	0 times
Fetches of paths robots.txt disallows	0
Observed	2026-08-19 to 2026-09-24

Measured on this site’s own traffic, verified against the operator’s published ranges. Every agent’s record: agent conduct.

Other agents from Anthropic

Claude-SearchBot — Claude search index
Claude-User — Claude live fetch
ClaudeBot — Anthropic training
Claude-Web — Anthropic legacy agent

See what anthropic-ai and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.

Questions about anthropic-ai

QHow do I block anthropic-ai in robots.txt?

QHow can I tell if a request from anthropic-ai is real?

Its operator publishes no IP ranges or DNS names for it, so a request carrying this name cannot be verified. Treat the name in a log line as a claim.

QDoes blocking anthropic-ai affect whether my site appears in AI answers?

No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.

QDoes CrawlCheck test my site as anthropic-ai?

Not as a request. CrawlCheck reads the rules your robots.txt sets for anthropic-ai and reports them. Rules for it are reported and never scored, because refusing it is a policy choice.