Glossary · Crawlers and access
user-triggered fetch exemption
The position several operators take that a fetch made because a user asked for a specific page, as ChatGPT-User or Claude-User do, is not governed by robots.txt, which addresses automated crawling. A conduct record therefore reports such an agent's fetches of disallowed paths without judging them.
What user-triggered fetch exemption means
The position several operators take that a fetch made because a user asked for a specific page, as ChatGPT-User or Claude-User do, is not governed by robots.txt, which addresses automated crawling. A conduct record therefore reports such an agent's fetches of disallowed paths without judging them. It can explain why a user-triggered agent reached a disallowed page; it cannot excuse a crawler that merely names itself as user-triggered, which the operator's published ranges settle.
Terms this definition uses
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about user-triggered fetch exemption
What is user-triggered fetch exemption?
The position several operators take that a fetch made because a user asked for a specific page, as ChatGPT-User or Claude-User do, is not governed by robots.txt, which addresses automated crawling. A conduct record therefore reports such an agent's fetches of disallowed paths without judging them.
What does user-triggered fetch exemption not show or guarantee?
The position several operators take that a fetch made because a user asked for a specific page, as ChatGPT-User or Claude-User do, is not governed by robots.txt, which addresses automated crawling. A conduct record therefore reports such an agent's fetches of disallowed paths without judging them. It can explain why a user-triggered agent reached a disallowed page; it cannot excuse a crawler that merely names itself as user-triggered, which the operator's published ranges settle.
Which area of the glossary does user-triggered fetch exemption belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← agent traffic index · retrieval crawler →
See it in the full glossary · 688 terms across 19 areas.