robots.txt, the way crawlers read it

Read groups as RFC 9309 does and spot a rule that binds the wrong agent.

robots.txt is read in groups: one or more User-agent lines, then the rules for them. A crawler obeys only the most specific group that names it, and ignores every other group, including *. So adding a group to say “GPTBot may read /blog/” quietly drops every Disallow you wrote under * for GPTBot, unless you repeat them.

Three more rules from the standard (RFC 9309) catch people out:

1. Within a group, the longest matching path wins; on a tie, Allow wins.
2. A blank line does not end a group, so a rule written after a blank line still belongs to the group above it. Some older parsers disagree, which is why the safest file keeps each group’s lines together.
3. A line after the last group belongs to that last group. A signal or rule meant for everyone, written at the bottom, binds only the last agent named.

Open your robots.txt. For every named agent, list which group it obeys and check that the rules you meant for everyone are repeated there.

← Lesson 1Lesson 3 →