Glossary · Crawlers and access
PerplexityBot vs Perplexity-User
The same split for Perplexity. PerplexityBot indexes for the answer engine and honours robots.txt; Perplexity-User fetches a page when a person's query requires it and, by Perplexity's own documentation, generally does not honour robots.txt because the request is treated as the user's. Both publish IP ranges. A site that sees Perplexity fetches after blocking PerplexityBot is usually seeing the second identity, not a violation by the first.
Terms this definition uses
Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · CCBot · Bytespider · agentic traffic · crawler classification
← ClaudeBot vs Claude-User · CCBot →
See it in the full glossary · 579 terms across 19 areas. Scan a site to see which of these apply to it.