Content-Signal
A robots.txt directive expressing separate permissions for search indexing, AI input and AI training.
What Content-Signal means
It communicates intent to operators who choose to honour it and carries no enforcement of its own.
Where Content-Signal comes up on this site
- Content-Signal in robots.txt: search, ai-input, ai-trainContent-Signal is still rare.
- Do AI crawlers and agents respect robots.txt?On one site the underlying file was excellent, with twenty-two agents allowed individually, a Content-Signal line, and a sitemap and llms.txt both declared.
- How to block AI crawlers like GPTBot and ClaudeBotThe Content-Signal guide covers which values to set.
- Meta robots vs X-Robots-Tag vs robots.txt: which one controls indexingThe newer AI signals, Content-Signal in robots.txt and the ai-train and ai-input vocabulary, decide what a fetched page may be used for.
- Naming a crawler in robots.txt drops your Disallow rulesFirst, a parser that closes a group at any unfamiliar line, such as Crawl-delay or Content-Signal , will attach the following rules to the wrong group; keep a group open across every directive until the next User-agent line that follows a rule.
Terms this definition uses
Definitions that name Content-Signal
tdmrep.json · TDM exception · per-use rights
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about Content-Signal
What is Content-Signal?
A robots.txt directive expressing separate permissions for search indexing, AI input and AI training.
Which area of the glossary does Content-Signal belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← AI opt-out · crawl budget →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.