CrawlCheck

Glossary · Machine files

TDM exception

Article 4 of the EU copyright directive of 2019: content may be mined for purposes including model training unless the rightsholder has expressly reserved that right in a machine-readable way. It is the legal thread that connects robots.txt directives, Content-Signal, ai.txt and tdmrep.json to something with force: a reservation expressed in one of them can matter in a European court. Whether a given operator reads any of them before fetching remains a policy question, and an expressed reservation is evidence of intent, not a guarantee of exclusion.

Terms this definition uses

Article · robots.txt · Content-Signal · ai.txt · tdmrep.json

Machine files

The files addressed to machines rather than readers, and what publishing one does and does not buy.

llms.txt · llms-full.txt · agents.md · entity map · entity map parity · machine layer · well-known URI · MCP · agent card · sitemap index · OpenAPI specification · ai.txt · Tool definition · SKILL.md · sitemap origin error · soft 404 on a machine file · tdmrep.json · security.txt

tdmrep.json  ·  security.txt

See it in the full glossary · 579 terms across 19 areas. Scan a site to see which of these apply to it.