AI crawlers · Training crawlers
GPTBot
GPTBot is the training crawler run by OpenAI: OpenAI training + index. Its operator publishes the IP ranges it crawls from (https://openai.com/gptbot.json), so a request carrying the name can be checked against that list.
| User-agent token | GPTBot |
| Operator | OpenAI |
| What it does | OpenAI training + index |
| Kind | training crawler |
| Can a request be verified? | Yes — published ip feed |
| Verification source | https://openai.com/gptbot.json |
| In a CrawlCheck scan | Sent as a request on every scan; reported, never scored |
Allow or block it in robots.txt
User-agent: GPTBot Disallow: /
User-agent: GPTBot Allow: /
Add a group that names it: "User-agent: GPTBot" followed by "Disallow: /". A crawler follows only the most specific group that names it, so once a group names GPTBot, rules under "User-agent: *" no longer apply to it; repeat in that group any Disallow lines you still want.
What blocking it changes
No. It collects pages for training. Refusing it is a policy choice about training, separate from whether answer engines can reach, read and quote the site.
What it actually did on crawlcheck.io
| Verified requests | 4,315 |
| Requests using its name from outside its operator's ranges | 136 (3.1% of claims) |
| robots.txt fetched | 0 times |
| Fetches of paths robots.txt disallows | 0 |
| Observed | 2026-08-12 to 2026-10-30 |
Measured on this site’s own traffic, verified against the operator’s published ranges. Every agent’s record: agent conduct.
Other agents from OpenAI
OAI-SearchBot— ChatGPT search indexChatGPT-User— ChatGPT live fetchOAI-AdsBot— OpenAI ChatGPT ads page validation
See what GPTBot and every other named agent receive from your own site: run a free scan. The full catalogue is on AI crawlers and as JSON at /ai-crawlers.json.