delivery comparison
Fetching the same URL as each named crawler identity and as a browser, then comparing status, redirect target and word count.
What delivery comparison does not show
It shows whether the site treats a crawler's name differently from a person; it cannot show whether the real crawler, arriving from its own address range, is treated the same way.
How CrawlCheck measures delivery comparison
4 finding codes measure it. Each one’s share of all scans is computed from the live corpus when this page is served, not typed into it.
| Finding code | Share of scans | Guide |
|---|---|---|
CRAWLER_SERVED_LESS | 1.1% | AI visibility tools compared: what each one measures |
CRAWLER_REDIRECTED_AWAY | 0.2% | AI crawlers refused, redirected or served less: how to check |
CRAWLER_SERVED_EXTRA | 1.4% | dataset |
ANSWER_ENGINE_REFUSED | 5.8% | Is Cloudflare blocking AI crawlers on your site? |
Every code, its rule and its current rate: the dataset page. Scan a site to see which apply to it.
Where delivery comparison comes up on this site
- HTTP status codes for crawlers: what a 200 can hideThe crawler delivery comparison guide covers refused, redirected and served-less in detail.
- Why word count cannot spot a parked domain: 783 wordsIf you are checking your own domain, fetch it from outside your network as a browser and as a crawler, and compare what comes back with what you built; the delivery comparison guide shows how.
- What AI crawlers receive from local business sitesThe delivery comparison guide covers all three.
Terms this definition uses
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · honeypot
Questions about delivery comparison
What is delivery comparison?
Fetching the same URL as each named crawler identity and as a browser, then comparing status, redirect target and word count.
What does delivery comparison not show or guarantee?
It shows whether the site treats a crawler's name differently from a person; it cannot show whether the real crawler, arriving from its own address range, is treated the same way.
How does CrawlCheck measure delivery comparison?
Through the finding codes CRAWLER_SERVED_LESS, CRAWLER_REDIRECTED_AWAY, CRAWLER_SERVED_EXTRA, ANSWER_ENGINE_REFUSED, each with a stated rule, evidence bytes and a live share of scans on the dataset page.
Which area of the glossary does delivery comparison belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← rate limit · honeypot →
See it in the full glossary · 792 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.