retrieval crawler
A crawler that fetches pages to answer a question being asked right now.
What retrieval crawler does not show
Distinct from a training crawler by purpose, not by capability, and the two are often operated by the same company under different user-agents.
Where retrieval crawler comes up on this site
- AI visibility tools compared: what each one measuresYour site what the server sends Retrieval crawler fetches and reads Answer named, cited or absent Supply side: machine-layer scanners, log verification Demand side: prompt monitors A prompt monitor only sees the right-hand box.
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about retrieval crawler
What is retrieval crawler?
A crawler that fetches pages to answer a question being asked right now.
What does retrieval crawler not show or guarantee?
Distinct from a training crawler by purpose, not by capability, and the two are often operated by the same company under different user-agents.
Which area of the glossary does retrieval crawler belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← user-triggered fetch exemption · user-triggered fetch →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.