Crawler trap
An unbounded set of generated URLs — faceted filters, infinite calendars, recursive relative paths — that a crawler can follow forever.
What Crawler trap does not show
It consumes crawl allocation without exposing new content.
Definitions that name Crawler trap
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about Crawler trap
What is Crawler trap?
An unbounded set of generated URLs — faceted filters, infinite calendars, recursive relative paths — that a crawler can follow forever.
What does Crawler trap not show or guarantee?
It consumes crawl allocation without exposing new content.
Which area of the glossary does Crawler trap belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← FCrDNS · Conditional request →
See it in the full glossary · 731 terms across 19 areas.