robots.txt for AI crawlers · free, no signup
robots.txt for GPTBot, ClaudeBot, PerplexityBot and Google-Extended: what yours actually permits
In one sentence
A robots.txt check for AI crawlers resolves the file per agent the way each crawler does — group by group, longest match first — and then compares that policy with what the edge actually served each identity, because an allow rule means nothing when the firewall answers 403.
Type a domain. The scan reads robots.txt once and resolves it per agent, the way each crawler would - group by group, longest match first - then compares that with what the edge actually served each identity. A rule that says allow means nothing if the firewall says 403.
What the scan measures for this
Per-agent policy
For every named AI crawler: allowed or refused at the root, whether it is named at all, and whether an allow-list for one agent silently shadowed the disallows meant for another.
The edge versus the file
What GPTBot, ClaudeBot and PerplexityBot actually received when they asked - a 200, a challenge page or a 403 - regardless of what robots.txt says.
The file itself
Status, content type, cache age and the Sitemap line, fetched as fifteen identities so a file that is served differently to a crawler is caught.
What the file points at
The Sitemap directive is followed: does it resolve, does it parse, and do the URLs it declares answer 200.
The workflow
- Scan the domainOne robots.txt fetch, resolved for every agent, plus the homepage as each identity so the edge is measured too.
- Read the AI agents sectionEach crawler on one row: named, allowed, shadowed, refused at the edge. The Fix list ranks the ones that hold the grade down.
- Download the corrected fileThe free fix below returns your robots.txt with the shadowed rules and the missing Sitemap line repaired, every change listed at the top.
- Publish, purge, re-scanA robots.txt is cached hard by most stacks. The second scan confirms the crawlers now read the file you meant to publish.
Free tools for this
Each one is a single request to the domain you name, answered as text or a download. Put the domain on the end of the URL, or use the form on the tools page.
Read next
- Do AI crawlers respect robots.txt?
- Eight crawlers, eight companies, one decision
- How to check if a GPTBot request in your logs is real
Start from what you are checking
You cannot pay to improve a score. A licence adds the lists behind the counts and the record over time - never a better grade.