Reach: who is knocking
Name the crawlers that matter and tell an edge block from a robots.txt rule.
An AI assistant does not see your site the way a visitor does. It sends a named crawler, and that crawler gets whatever your server, CDN or firewall decides to give it. Three jobs share the work, and each has its own name: training crawlers collect pages to train models (GPTBot, ClaudeBot, CCBot); search crawlers build the index an assistant answers from (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot); user agents fetch one page live because someone asked about it (ChatGPT-User, Claude-User, Perplexity-User). Google-Extended and Applebot-Extended are different again: they never fetch anything. They are names you write in robots.txt to say whether content Googlebot or Applebot already fetched may be used for AI.
A crawler can be stopped in two places. robots.txt asks it politely, and a well-behaved crawler obeys. Your edge (a CDN bot rule, a firewall, a “block AI bots” switch) refuses the request outright, usually with a 403 or a challenge page, while browsers still get a normal 200. Owners often have no idea the second one is on.
On your own site
Scan your site at crawlcheck.io. In the Reach grid, find any crawler that got a different answer from the browser control. For each one, write down whether robots.txt or the edge stopped it.