What each AI is served from your homepage
The same page, asked for by two browsers and by requests carrying the names of ChatGPT, Claude, Perplexity, Google, Bing and Apple’s crawlers, side by side. Then one test no other checker runs: does your firewall let a request in just because it says it is a crawler?
Not needed: a plain browser was served
A plain browser was served, so there was no block for a name to get past.
10 of 10 agent names served · 2 of 2 browsers
| Request | Result | Words | Prices seen | Links in / out | robots.txt on / |
|---|---|---|---|---|---|
| Chrome (desktop) | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | n/a |
| Chrome (mobile) | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | n/a |
| Request claiming to be ChatGPT-User OpenAI · fetches a page when a ChatGPT user asks | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | disallowed (named) |
| Request claiming to be OAI-SearchBot OpenAI · builds ChatGPT search | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | disallowed (named) |
| Request claiming to be GPTBot OpenAI · collects training data | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | disallowed (named) |
| Request claiming to be Claude-User Anthropic · fetches a page when a Claude user asks | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | disallowed (named) |
| Request claiming to be Claude-SearchBot Anthropic · builds Claude search | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | disallowed (named) |
| Request claiming to be ClaudeBot Anthropic · collects training data | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | disallowed (named) |
| Request claiming to be PerplexityBot Perplexity · builds Perplexity search | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | disallowed (named) |
| Request claiming to be Googlebot Google · Google Search, AI Overviews and AI Mode | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | allowed |
| Request claiming to be Bingbot Microsoft · Bing and Copilot | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | allowed |
| Request claiming to be Applebot Apple · Siri, Spotlight and Apple Intelligence | served | 2101 | $3.7billion, $60,000, $22million, $15.8M | 202 / 26 | disallowed (named) |
- Google-Extended (Google, Gemini training and grounding) is a robots.txt name only, never a request: robots.txt disallows it.
- Applebot-Extended (Apple, Apple model training) is a robots.txt name only, never a request: robots.txt disallows it.
Where the block happens
Nothing was refused from CrawlCheck's network, so there is no block to place.
No own-connection reading in the last 14 days. Install the CrawlCheck extension, open this site and press Two vantages: a block that only refuses data-centre addresses shows up here. Own this site? Let CrawlCheck past your firewall with one skip rule, and the next reading says whether a refusal is that firewall or your server.
Each row is a request claiming to be the named agent, sent once to the homepage from CrawlCheck's network. A user-agent string is a name tag anyone can type; the real crawlers come from their operators' networks and are verified by IP range or Web Bot Auth. This shows what a site does with the name alone. Reported, never scored. Signed JSON: /api/v1/agent-lens?domain=denverpost.com.
How to read it
- A name tag is not an identity. Every row below is a request claiming to be that agent. The real crawlers come from their operators’ networks and prove it by IP range or a Web Bot Auth signature; a site that checks properly can serve this request differently from the real one.
- The name-tag test only fires when both browsers are refused and a request that merely claims a crawler’s name is let in, twice in a row. That means the gate decides on a string anyone can type.
- Comparison only. One request per persona to the homepage; nothing behind a login, no forms, no second page. A refused persona is reported, never retried under another name to get in.
- Not scored. This page reports; the grade on a full scan is unchanged by it. For the signed answer agents act on, see Agent view.