CrawlCheck

Agent view · Guides · 2026-10-03

How to test robots.txt for GPTBot and ClaudeBot: 5 tools compared

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/how-to-test-robots-txt-for-gptbot-and-claudebot: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/how-to-test-robots-txt-for-gptbot-and-claudebot>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes22,713
words3,369
sha-2567409ccb8e1f6e181ef2bc6965a276a13fe0bf65ef8e94000e1f7d1da2ac86570
repr-digestsha-256=:dAnMuOH24YHvK8aWWidqE/4L9l746UAA4ffR2irIZXA=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311495;expires=1795315095;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="VGRuldNIynEGIg-mN9HGAg";tag="crawlcheck-page"
signaturesig1=:nlvE0n+8Do1X2Mh6fjMvoyiF2mb9BEkkj3q9xd/R7UPL8CRRNvFFP4Tg/skAN764RVRfVunnFTFCCyiUE6FxCg==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-10-03 · By [VSNARY | Emmanuel Orta](/author) · 4 views

# How to test robots.txt for GPTBot and ClaudeBot: 5 tools compared

Policy testers tell you whether your file allows a crawler; only a delivery check tells you whether the crawler gets the page. Five tools compared, the two-minute test, and the mistakes we see most.

To test robots.txt for GPTBot or ClaudeBot, run your live file through a checker that accepts any user-agent; CrawlCheck's free checker resolves it for every AI crawler on its list at once. Then check delivery: fetch the page as that crawler, because a firewall or CDN can refuse it even when robots.txt allows it. Google Search Console reports only Google's own fetch, and Google's open-source parser tests the file but not delivery.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- What does testing robots.txt for GPTBot actually involve?
- The tools compared
- How does a crawler choose which rules apply to it?
- How to test robots.txt for GPTBot in two minutes
- How often do these problems occur?
- What do status codes on robots.txt do?
- Which AI crawlers should you test?
- The mistakes we see most
- When should you test again?
- Where CrawlCheck is weaker
- Summary

Testing [robots.txt](/glossary/robots-txt) for AI crawlers takes two checks, and most tools do only the first. The first is **policy**: given your file, is GPTBot (or ClaudeBot, or PerplexityBot) allowed to fetch a given [URL](/glossary/url)? The second is **delivery**: when that crawler actually asks, does your server or [CDN](/glossary/cdn) serve the page, or refuse it regardless of what robots.txt says? A file that allows GPTBot, behind a firewall that answers it with a 403, reads as “allowed” in every policy tester.

**Disclosure:** CrawlCheck publishes this comparison and is one of the products in it. Every figure about another company comes from that company’s own pricing or documentation page, linked where it appears, read on 3 October 2026. Where CrawlCheck does less than a competitor, the tables say so.

**Key points**

- A robots.txt test has two parts: policy (what the file allows) and delivery (what the crawler is actually served).
- Under RFC 9309 a crawler named in its own group ignores the `User-agent: *` group entirely, so adding an AI crawler’s group can silently drop your Disallow rules.
- A 4xx on robots.txt means “no rules”; a 5xx means “disallow everything” while it lasts.
- Across CrawlCheck’s scans, robots.txt allows an [answer engine](/glossary/answer-engine)’s crawler while the edge refuses it on 5.8% of sites.
- CrawlCheck’s free checker resolves the live file for every AI crawler on its list at once; Google’s tools cover Google’s own crawlers.

## What does testing robots.txt for GPTBot actually involve? #

Most robots.txt testers stop at check 1. The crawler lives with the result of check 2.
Policy and delivery are decided in different places. The file lives on your server and is read by the crawler. Delivery is decided before that, by whatever answers the request first: a CDN, a firewall, a security plugin, a hosting provider’s bot filter. When those layers refuse a crawler, robots.txt is never consulted, and no policy tester can know, because a policy tester only reads the file.

## The tools compared #

| Tool | Tests any user-agent (GPTBot, ClaudeBot)? | Checks delivery, not just the file? | Cost |
| --- | --- | --- | --- |
| **CrawlCheck robots.txt checker** | Yes: resolves the live file for every AI crawler on its list at once, group by group, most specific match first | Yes: the full scan compares the policy with what each crawler identity was actually served | Free, no signup |
| Google Search Console robots.txt report | Shows the robots.txt files Google found, when they were crawled, and parsing warnings or errors | No; it reports Google’s own fetch of the file | Free, verified site owners |
| Google’s open-source robots.txt parser | Yes: you pass any user-agent and URL | No; it evaluates the file only | Free; needs a developer to build and run |
| TechnicalSEO.com robots.txt tester (Max Prin) | Tests whether a URL is blocked and how | No | Free web tool |
| Cloudflare AI Crawl Control | Tracks robots.txt health and which crawlers violate your directives, on a Cloudflare zone | Sees requests that reach your zone; does not test from outside | Included on Cloudflare plans |

Sources: [Google’s robots.txt report help page](https://support.google.com/webmasters/answer/6062598), [Google’s robotstxt library](https://github.com/google/robotstxt), [TechnicalSEO.com](https://technicalseo.com/tools/robots-txt/), [Cloudflare AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/).

## How does a crawler choose which rules apply to it? #

A robots.txt file is a list of groups. Each group starts with one or more `User-agent` lines and carries `Allow` and `Disallow` rules. RFC 9309 sets three rules for reading it that every AI crawler test should apply:

- **A crawler uses the group that names it.** If several groups name it, their rules are combined. Only when no group names it does it fall back to the group for `*`.
- **Within that group, the most specific rule wins,** meaning the matching rule with the longest path. If an Allow and a Disallow match equally, the Allow should be used.
- **Rules are not inherited.** A crawler with its own group does not also obey the `*` group.

This is the most common robots.txt surprise we measure: adding a group to allow one AI crawler silently drops the Disallow rules written for everyone. Under RFC 9309, a crawler uses the group that names it, and the * group only when no group does.
The third rule causes most real-world damage. An owner adds `User-agent: GPTBot` with `Allow: /blog/` to welcome OpenAI’s crawler, and in doing so removes every Disallow written under `*` for GPTBot: the cart, the account pages, the staging paths. Nothing errors and nothing looks different in a browser. The fix is to repeat the shared Disallow lines inside each named group. Our own robots.txt work hit this exact trap; the full explanation is in [why naming a crawler drops your Disallow rules](/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules).

## How to test robots.txt for GPTBot in two minutes #

- **Open the checker.** Go to [robots.txt for AI crawlers](/robots-txt-for-ai-crawlers) and enter your domain, plus a path if you care about a page other than the homepage.
- **Read the per-crawler result.** Each AI crawler is listed as allowed or disallowed for that path, grouped by role: answer engines, search, training.
- **Check the shadowed groups.** The checker lists named groups that silently drop rules from the `*` group.
- **Check the file itself.** Run `curl -sI https://example.com/robots.txt` and confirm status 200 and `content-type: text/plain`. An HTML page at that address is not a robots.txt file, whatever its status.
- **Run the full scan.** From [the homepage](/), scan the domain to see whether each crawler is actually served the page your file permits.

## How often do these problems occur? #

robots.txt and access problems on scanned sites — live share of every CrawlCheck scan, read when this page loaded
robots.txt allows, the edge refuses 5.8%

Named groups drop the * rules 14%

An AI opt-out already set 1.8%

robots.txt itself refused or erroring 0.4%

A policy tester catches the second and third rows. Only a delivery check catches the first, and only a fetch of the file itself catches the fourth.

## What do status codes on robots.txt do? #

The response to the robots.txt request itself changes everything that follows, and the rules are not intuitive:
From RFC 9309. A 403 on robots.txt means “no rules”, not “keep out”; a 503 means “keep out” for as long as it lasts. Neither is what most owners intend, so serve the file with status 200 and content type text/plain.
Two cases cause most trouble. A security layer that answers 403 to unknown clients on every path, including `/robots.txt`, tells a crawler there are no rules, and then refuses the pages anyway, so the crawler gets nothing and the owner’s file is never read. A server under load that answers 503 tells every crawler to stay away completely until it recovers. Detail and examples: [robots.txt status codes: what 404, 403 and 503 tell a crawler](/guides/robots-txt-status-codes-what-404-403-and-503-tell-a-crawler).

## Which AI crawlers should you test? #

Each name is a separate robots.txt decision. Blocking a training crawler does not remove a site from that company’s search answers; user-initiated fetchers act for a person, and operators treat robots.txt differently for them, so read each operator’s documentation.
Test each one you care about separately, because each is controlled separately. OpenAI documents GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for pages a person asks ChatGPT to open. Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User in the same three roles. Perplexity documents PerplexityBot for its answers and Perplexity-User for user requests. Disallowing GPTBot does not remove a site from ChatGPT search; that is OAI-SearchBot. The full list, with each operator’s own documentation, is on [the AI crawler list](/ai-crawlers).

## The mistakes we see most #

| Mistake | What happens | How to spot it |
| --- | --- | --- |
| robots.txt allows the crawler, the firewall refuses it | The crawler never reads the page; policy testers say “allowed” | Fetch the page as the crawler and as a browser and compare |
| A named group without the shared rules | Disallow lines meant for everyone stop applying to that crawler | The checker’s shadowed-groups list |
| robots.txt answering 403 or 503 | 4xx reads as no rules; 5xx as full disallow | `curl -sI` on the file |
| robots.txt served as HTML | No directives can be parsed, so no rules apply | The `content-type` header |
| Blocking the training crawler to leave search | The site stays in, or drops out of, the wrong product | Check each operator’s crawler roles |
| Editing the file and testing a cached copy | The test reads the new file; crawlers get the old one | Fetch the bare URL and a cache-busted URL and compare |

## When should you test again? #

Re-test after any edit to robots.txt, after installing or updating an SEO or security plugin (either can rewrite the file or add bot rules), after moving host or CDN, and after switching on any bot-protection feature. Under RFC 9309 crawlers should not use a cached robots.txt for more than 24 hours, so a change can take up to a day to reach them, and a cached copy at your own edge can hold it back longer. Test the bare URL, not a cache-busted one, because the bare URL is what crawlers request.

**Keep an acceptance record after a fix.** Save the exact rule before and after, the target URL, the time of each response, the user-agent and the network you tested from. Inspect the body, not just the status: a 200 can carry a challenge or login page. Then treat citations and referrals as separate measurements: a crawler that can now fetch the page has not thereby cited it. The limits of what a policy test proves are set out in [what an access repair proves, and what it does not](/blog/unblocking-ai-crawlers-access-is-not-a-citation-result).

## Where CrawlCheck is weaker #

CrawlCheck sends each crawler’s published user-agent but cannot send from OpenAI’s or Anthropic’s own addresses, so a firewall rule keyed only to those IP ranges is reported as a difference between identities rather than confirmed. It reads the live file on your domain and does not let you test a draft before publishing; Google’s open-source parser does that, if you can build it. Search Console, unlike CrawlCheck, shows Google’s own fetch history for your file, so for Googlebot use it alongside.

## Summary #

A policy tester answers “does my file allow GPTBot?”. Only a delivery check answers “does GPTBot actually get the page?”. Run both: the free [robots.txt checker](/robots-txt-for-ai-crawlers) for the policy across every AI crawler, then a scan for delivery. More: [do AI crawlers respect robots.txt?](/guides/do-ai-crawlers-respect-robots-txt)

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-test-robots-txt-for-gptbot-and-claudebot%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-test-robots-txt-for-gptbot-and-claudebot%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=How%20to%20test%20robots.txt%20for%20GPTBot%20and%20ClaudeBot%3A%205%20tools%20compared)[Email](mailto:?subject=How%20to%20test%20robots.txt%20for%20GPTBot%20and%20ClaudeBot%3A%205%20tools%20compared&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhow-to-test-robots-txt-for-gptbot-and-claudebot)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### How do I test robots.txt for GPTBot?

Enter your domain in a robots.txt checker that accepts any user-agent, such as CrawlCheck's free robots.txt checker, which resolves the live file for every AI crawler on its list, including GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Then confirm delivery with a scan, because a firewall can refuse GPTBot even when robots.txt allows it.

### Does Google Search Console test robots.txt for GPTBot?

No. Search Console's robots.txt report shows the files Google found for your site, when they were crawled, and any warnings or errors. It reports Google's own fetch, not other companies' crawlers.

### Why does my robots.txt allow GPTBot but ChatGPT still cannot read my site?

Usually a firewall, bot-management or CDN rule refuses the crawler before robots.txt matters, or the page serves a challenge. A delivery check that fetches as GPTBot shows the actual response.

### If I add a group for GPTBot, does it still follow my User-agent: * rules?

No. Under RFC 9309 a crawler uses the group that names it and ignores the * group, so any Disallow rules you want GPTBot to obey must be repeated inside its own group.

### Which rule wins when Allow and Disallow both match?

The most specific match, meaning the rule with the longest matching path. If an Allow and a Disallow match equally, RFC 9309 says the Allow should be used.

### Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI documents GPTBot for training and OAI-SearchBot for ChatGPT search results. Each is controlled by its own robots.txt group.

### What happens if robots.txt returns 403 or 503?

Under RFC 9309, a 4xx response means crawlers may treat the file as absent and fetch anything, and a 5xx means they must assume complete disallow while it lasts. Serve robots.txt with status 200 and content type text/plain.

### Can I test robots.txt rules without publishing them?

Google's open-source robotstxt parser evaluates any file, user-agent and URL locally if you can build and run it. CrawlCheck's checker reads the live file on your domain.

### What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects pages for training OpenAI's models. OAI-SearchBot fetches pages for ChatGPT search, and ChatGPT-User fetches a page when a person asks ChatGPT to open it. Each follows its own robots.txt group, so you can block GPTBot and still allow OAI-SearchBot. OpenAI says sites that disallow OAI-SearchBot are not shown in ChatGPT search answers, and that a robots.txt change takes about 24 hours to apply to search.

### How do I check server logs for GPTBot requests?

Filter the access log for the user-agent string GPTBot, then check each request's IP address against the ranges OpenAI publishes at openai.com/gptbot.json (searchbot.json and chatgpt-user.json cover the other two). A request that names GPTBot from an address outside those ranges is not OpenAI. The status code on each line shows what the crawler actually received.

### How do I change robots.txt on WordPress?

WordPress serves a generated robots.txt when there is no file in the site's root folder. Upload a real robots.txt to the root and it replaces the generated one; many SEO plugins also include a robots.txt editor. Then fetch the live /robots.txt to confirm the change, because a caching plugin or CDN can keep serving the old version.

## Related findings

[2026-08-28 How to check if AI crawlers can read your site Five curl checks that show whether GPTBot, ClaudeBot and PerplexityBot can read your site: content type, cache, crawler vs browser](/guides/how-to-check-if-ai-crawlers-can-read-your-site)[2026-10-15 Robots.txt allows it, the server refuses: 57 top sites At 57 of 418 top sites, the server refused GPTBot, ClaudeBot, OAI-SearchBot or PerplexityBot even though robots.txt allows it. How](/blog/robots-txt-allows-server-refuses-ai-crawlers-october-2026)[2026-10-03 Verify GPTBot, ClaudeBot and PerplexityBot: tools compared How to verify GPTBot, ClaudeBot and PerplexityBot requests: published IP lists, CrawlCheck's free log check and Cloudflare verifie](/guides/how-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [/](/) · [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) | **Glossary:** [MCP transport](/glossary/mcp-transport) · [cite-as link](/glossary/cite-as-link) · [x-default](/glossary/x-default) | **Registry:** [astronomy.com](/registry/astronomy.com) · [deckardlab.tech](/registry/deckardlab.tech) · [hp.com](/registry/hp.com) · [netangels.ru](/registry/netangels.ru) · [think.resoneo.com](/registry/think.resoneo.com)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/how-to-test-robots-txt-for-gptbot-and-claudebot.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.