AI crawlers · SEO and research crawlers
Cloudflare-AI-Search-External
Cloudflare-AI-Search-External is the tool or developer fetch run by Cloudflare: Cloudflare AI Search External. Verify the HTTP Message Signature against the operator’s public key directory: https://search.ai.cloudflare.com/external/.well-known/http-message-signatures-directory.
| User-agent token | Cloudflare-AI-Search-External |
| Operator | Cloudflare |
| What it does | Cloudflare AI Search External |
| Kind | tool or developer fetch |
| Can a request be verified? | Yes — web bot auth |
| Verification source | https://search.ai.cloudflare.com/external/.well-known/http-message-signatures-directory |
| Ready-made robots.txt | Block training and corpus crawlers · Block every AI crawler · Block SEO and ad crawlers · server rules · this page as JSON |
| In a CrawlCheck scan | Read from robots.txt; reported, never scored |
Allow or block it in robots.txt
User-agent: Cloudflare-AI-Search-External Disallow: /
User-agent: Cloudflare-AI-Search-External Allow: /
Add a group that names it: "User-agent: Cloudflare-AI-Search-External" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names Cloudflare-AI-Search-External, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.
What blocking it changes
No. It is a third-party tool's crawler. Blocking it changes what that tool knows about your site, not anyone's AI answers.
Other agents from Cloudflare
CloudflareBrowserRenderingCrawler— Cloudflare Browser Run /crawlCloudflare-AutoRAG— Cloudflare AutoRAG
See what Cloudflare-AI-Search-External and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.