OAI-SearchBot vs PerplexityBot
In the census run that finished 2026-10-24, robots.txt closed the homepage to OAI-SearchBot on 280 of the 4,501 top sites that serve a robots.txt (6.2%), and to PerplexityBot on 409 (9.1%).
| OAI-SearchBot | PerplexityBot | |
|---|---|---|
| Operator | OpenAI | Perplexity |
| What it is | answer-engine crawler | answer-engine crawler |
| What it does | ChatGPT search index | Perplexity index |
| How a request can be verified | Its operator publishes the IP ranges it crawls from (https://openai.com/searchbot.json), so a request carrying the name can be checked against that list. | Its operator publishes the IP ranges it crawls from (https://www.perplexity.ai/perplexitybot.json), so a request carrying the name can be checked against that list. |
| robots.txt closes the homepage (of sites with a robots.txt) | 280 (6.2%) | 409 (9.1%) |
| Refused at the door (of sites that answered a browser) | 702 (12.6%) | 733 (13.2%) |
Top 10,000 sites, census of 2026-10-24. “Refused at the door” is a non-2xx answer to a client declaring that name, from CrawlCheck’s own addresses, where the browser got 2xx. Every crawler on the radar.
Let OAI-SearchBot in and keep PerplexityBot out
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Disallow: /A crawler obeys only the most specific group that names it and ignores * entirely. If your User-agent: * group has Disallow rules, repeat them under OAI-SearchBot, or naming it drops them. Swap the two names to do the reverse.
Check what your own site tells both: scan it free. Catalogue data CC BY 4.0 at /ai-crawlers.json.