Findings · 2026-10-30 · By VSNARY | Emmanuel Orta
Unblocking AI crawlers: what an access repair proves
Twelve synthetic robots.txt cases show that the agent’s name decides which rules apply. They do not show access, citations or referrals, and this note says so.
An access repair proves only what it measured. In twelve synthetic cases run through Python's robots.txt parser, naming a crawler changed which group of rules applied, exactly as the standard says. That is a policy result. It does not show that the real crawler received the page, that an answer engine cited it, or that anyone clicked through. After a real fix, keep a before-and-after fetch as the crawler, and measure citations and referrals separately.
1/4
The fixtures measure one event: which robots.txt group applies to a named agent. They do not measure delivery, a citation, or a referral. Those are the other three.
What this note is. On 30 October 2026 we ran twelve synthetic robots.txt cases through Python’s standard-library parser: four policy strings, twelve agent-and-path combinations. All twelve returned the expected result. The result is narrow, and this note says exactly how narrow, because a robots.txt fix is easy to oversell as an AI-visibility result.
What did the twelve cases show? #
One thing: the name of the agent decides which robots.txt group applies. Nine cases test a general restriction against a named-agent group. Three test a file that blocks a training token (GPTBot) while leaving a search token (OAI-SearchBot) allowed.
| Policy | Client and path | Returned permission |
|---|---|---|
| General group blocks /private/ | GPTBot /private/a | Denied |
| Named GPTBot group allows / | GPTBot /private/a | Allowed |
| Named GPTBot group repeats the restriction | GPTBot /private/a | Denied |
| Training denied, search allowed | OAI-SearchBot /private/a | Allowed |
The second row is the trap site owners walk into: naming a crawler to welcome it removes the restrictions written for everyone. The fourth is a reminder that a path that looks private is not private; only authentication is. The full explanation is in why naming a crawler drops your Disallow rules.
What did they not show? #
| What an owner or agency wants to know | Measured here? |
|---|---|
| Which policy group applies depends on the agent name | Yes, on the supplied cases |
| A named crawler received the page from the live site | No request was sent |
| An answer engine cited the page | No answer was collected |
| A person followed the citation | Not in this design |
Synthetic inputs, one parser, no network. A file another parser reads differently, or a CDN serving the crawler an older file, are real problems this did not measure. Quoting these cases as proof that unblocking “works” borrows a policy result to pay for a citation claim.
What should a site owner take from it? #
Test the file you actually deployed, with the token you actually meant, and then fetch the live page as that crawler. A fix that lets the crawler in is a delivery result; citations and referrals are separate measurements, on their own clocks. If access improves and citations do not, report both. The step-by-step test, with an acceptance record to keep after a fix, is in how to test robots.txt for GPTBot and ClaudeBot; the live gap between policy and delivery is in robots.txt says yes, the server says no.
The cases, expected results and returned results are in the dated evidence bundle. References: RFC 9309, OpenAI’s crawler documentation, Google’s AI features guidance, which also separates technical eligibility from any guarantee of inclusion.
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Questions this post answers
Does unblocking AI crawlers get my site cited?
Not by itself. Unblocking lets a crawler fetch the page. Whether an answer engine cites it, and whether anyone clicks through, are separate outcomes that need their own measurement.
What did the twelve robots.txt cases prove?
That the agent name decides which robots.txt group applies, in one parser and on the supplied cases. They did not send any request to a live site or collect any AI answer.
Why does naming GPTBot in robots.txt remove my other rules?
A crawler obeys only the group that names it and ignores the group for all agents. Any restriction you want it to keep must be repeated inside its own group.
What should I record after fixing robots.txt?
The exact rule before and after, the target URL, the response time, the user-agent and the network tested from, and the body of the response, because a 200 can carry a challenge page. Measure citations and referrals separately afterwards.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.