1 · See it
The agent view
The answer an agent receives about any domain, rendered for a person: each action’s decision and the reason.
robots.txt for GPTBot, ClaudeBot, PerplexityBot and Google-Extended: what yours actually permits
In one sentence
A robots.txt check for AI crawlers resolves the file per agent the way each crawler does — group by group, longest match first — and then compares that policy with what the edge actually served each identity, because an allow rule means nothing when the firewall answers 403.
Type a domain. The scan reads robots.txt once and resolves it per agent, the way each crawler would - group by group, longest match first - then compares that with what the edge actually served each identity. A rule that says allow means nothing if the firewall says 403.
Measured
What the scan measures for this
Per-agent policy
For every named AI crawler: allowed or refused at the root, whether it is named at all, and whether an allow-list for one agent silently shadowed the disallows meant for another.
The edge versus the file
What GPTBot, ClaudeBot and PerplexityBot actually received when they asked - a 200, a challenge page or a 403 - regardless of what robots.txt says.
The file itself
Status, content type, cache age and the Sitemap line, fetched as fifteen identities so a file that is served differently to a crawler is caught.
What the file points at
The Sitemap directive is followed: does it resolve, does it parse, and do the URLs it declares answer 200.
For agents
What an agent is told about the same site
An agent does not read the report. It asks one endpoint before it reads, cites, connects or transacts, and gets a signed answer built from the same measurement.
2 · Call it
One endpoint
GET /api/v1/resolve?domain= returns the signed answer; preflight returns a decision and a receipt for one action.
3 · Check it
Without trusting us
Every answer is Ed25519-signed and logged; it verifies offline with the published key.
Workflow
The workflow
- Scan the domainOne robots.txt fetch, resolved for every agent, plus the homepage as each identity so the edge is measured too.
- Read the AI agents sectionEach crawler on one row: named, allowed, shadowed, refused at the edge. The Fix list ranks the ones that hold the grade down.
- Download the corrected fileThe free fix below returns your robots.txt with the shadowed rules and the missing Sitemap line repaired, every change listed at the top.
- Publish, purge, re-scanA robots.txt is cached hard by most stacks. The second scan confirms the crawlers now read the file you meant to publish.
Free
Free tools for this
Each one is a single request to the domain you name, answered as text or a download. Put the domain on the end of the URL, or use the form on the tools page.
Resolve robots.txt per agentAllowed, allowed at the root, named, shadowed - for every AI crawler, from one fetch. Add &path=/some/page to test a path.Download your robots.txt, correctedYour file with the shadowed rules and the missing Sitemap line repaired; every change listed at the top.
Reading
Read next
- Do AI crawlers and agents respect robots.txt?
- Eight crawlers, eight companies, one decision
- How to check if a GPTBot request in your logs is real
Start from what you are checking
llms.txt auditdoes yours resolve, parse and match the site it describes?Entity map validatordoes your entitymap.json parse, and does every claim in it resolve?SEO audit toolthe technical checks, plus what AI crawlers actually receiveTechnical SEO auditcrawl access, sitemap, hosts, links and speed, checked in one scanLocal SEO auditdoes your site say who you are, where, and does every listing agree?SEO audit reports for agenciesyour name on the report, the proof still attachedWhat is a website auditWhat is a website audit? What it checks, what it costs, and how to run one freeWhat is an SEO auditWhat is an SEO audit? The checks that matter, in the order to fix themAEO auditcan ChatGPT, Claude, Perplexity and Google reach, read and quote your site?
You cannot pay to improve a score. A licence adds the lists behind the counts and the record over time - never a better grade.
Questions about this page
QMy robots.txt allows GPTBot. Why is it marked refused?
Because robots.txt is a request and the edge is a decision. If the firewall, bot-fight mode or a challenge page answers the crawler with anything but 200, the crawler never reads the page, whatever the file says. The Trust section shows the bytes each identity received.
QWhat is a shadowed rule?
A group written for one user-agent that silently voids the rules meant for another. The most common form is an allow-list added for named AI agents that made every Disallow above it stop applying to them. The per-agent resolver names which agent each rule actually reaches.
QDo AI crawlers obey robots.txt?
The major operators publish that their crawlers do, and verified requests in our own logs are consistent with that. Unverified traffic wearing a crawler's name is a different matter; the free verifier checks a log line against published address ranges.
QShould I block AI crawlers?
That is a business decision the scan does not make for you. It records what each crawler is told and what it receives, so the decision you made is the one actually in force.