AI opt-out
Any published instruction asking that a site's content not be used for model training.
What AI opt-out does not show
It is a request rather than a control, and whether it is honoured varies by operator.
How CrawlCheck measures AI opt-out
One finding code measures it. Its share of all scans is computed from the live corpus when this page is served, not typed into it.
| Finding code | Share of scans | Guide |
|---|---|---|
AI_OPTOUT_SET | 4.2% | Is Cloudflare blocking AI crawlers on your site? What changes on 15 September 2026 |
Every code, its rule and its current rate: the dataset page. Scan a site to see which apply to it.
Where AI opt-out comes up on this site
- Is Cloudflare blocking AI crawlers on your site? What changes on 15 September 2026That is the general shape of it: ANSWER_ENGINE_REFUSED fires on of the scans in the public dataset, and find an AI opt-out already set.
- How to rank in ChatGPT: the part that is actually in your controlSeparately, of scans find an AI opt-out already set .
- GEO, AEO, LLM SEO and the agentic web: what the terms mean and which parts can be measuredANSWER_ENGINE_REFUSED fires on of scans, and carry an AI opt-out — much of it inherited rather than chosen, given that the eight largest opt-out groups on the web sit within 12% of each other.
- Best AI web crawler on a budget: real costs comparedWhat none of them is built to do, and what CrawlCheck exists for, is fetch it once per AI crawler identity, compare what each one was served against a browser, read robots.txt and the AI opt-out signals per crawler, and tell you which of those differences…
- Agent readiness checkers: Is It Agent Ready? vs CrawlCheckAcross every site CrawlCheck has scanned, read live: an answer engine’s crawler is refused while a browser is served on ; robots.txt groups named for AI crawlers silently drop the * rules on ; an AI opt-out is already set on .
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about AI opt-out
What is AI opt-out?
Any published instruction asking that a site's content not be used for model training.
What does AI opt-out not show or guarantee?
Any published instruction asking that a site's content not be used for model training. It is a request rather than a control, and whether it is honoured varies by operator.
How does CrawlCheck measure AI opt-out?
Through the finding code AI_OPTOUT_SET, each with a stated rule, evidence bytes and a live share of scans on the dataset page.
Which area of the glossary does AI opt-out belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← user-agent group · Content-Signal →
See it in the full glossary · 731 terms across 19 areas.