Content Signals
A robots.txt extension proposed in 2025 in which a site states, per path, whether its content may be used for search, for AI input at answer time, and for AI training, as three separate signals.
What Content Signals does not show
It is a declaration crawlers may read; nothing enforces it, and a crawler that ignores robots.txt ignores this too.
Where Content Signals comes up on this site
- Is Cloudflare blocking AI crawlers on your site?Enterprise Bot Management customers can declare how bots may use content after crawling it, through a use parameter that extends the Content Signals vocabulary in robots.txt.
- How to block AI crawlers like GPTBot and ClaudeBotA use parameter extends the Content Signals vocabulary in robots.txt, and compliance by verified bots is reported rather than enforced.
- Content-Signal in robots.txt: search, ai-input, ai-trainContent signals state site-wide preferences for three categories of use.
- Markdown for AI agents: serve text/markdown by negotiationThat default may not match your robots.txt, so if your policy differs, send your own header from the origin; the values are explained in the content signals guide .
- How an AI agent should verify a website before it actsA site declares itself in several places: agent manifests , robots.txt and its Content Signals, llms.txt, structured data, an A2A agent card, MCP listings.
Terms this definition uses
Definitions that name Content Signals
Related terms in Machine files
The files addressed to machines rather than readers, and what publishing one does and does not buy.
llms.txt · llms-full.txt · agents.md · entity map · entity map parity · machine layer · well-known URI · MCP · agent card · sitemap index · OpenAPI specification · ai.txt · Tool definition · SKILL.md · sitemap origin error · soft 404 on a machine file · tdmrep.json · TDM exception · security.txt · HTTP Message Signatures · changefreq and priority · staging leak · host duplicate
Questions about Content Signals
What is Content Signals?
A robots.txt extension proposed in 2025 in which a site states, per path, whether its content may be used for search, for AI input at answer time, and for AI training, as three separate signals.
What does Content Signals not show or guarantee?
It is a declaration crawlers may read; nothing enforces it, and a crawler that ignores robots.txt ignores this too.
Which area of the glossary does Content Signals belong to?
Machine files: The files addressed to machines rather than readers, and what publishing one does and does not buy.
← security.txt · HTTP Message Signatures →
See it in the full glossary · 792 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.