GPTBot vs OAI-SearchBot

Two OpenAI crawlers with one operator and different purposes. GPTBot collects for model training; OAI-SearchBot fetches for live search retrieval and is what a ChatGPT search citation depends on; a third identity, ChatGPT-User, fetches when a person asks about a URL. Each has its own range in OpenAI's published feed and its own robots.txt token. Blocking one says nothing about the others, which is why allow and block decisions are per agent, not per company, and why a rule addressed to GPTBot alone leaves search retrieval untouched.

Terms this definition uses

retrieval · citation · robots.txt · token

Crawlers and access

Who is fetching, whether they are who they claim, and what your rules actually permit.

retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification

← Google-Agent · ClaudeBot vs Claude-User →

See it in the full glossary · 579 terms across 19 areas. Scan a site to see which of these apply to it.