Who sits between AI agents and websites
Month to date (2026-10), rebuilt after each census run; sealed after the month ends. Every reachable site in the top 10,000, weekly census frame (run 2, 2026-10-10), placed in the four layers an AI crawler’s request passes through, with what the site did to AI crawlers. No site is named.
DNS host
who answers the domain's DNS (NS records). 27 named.
| DNS host | Sites | Edge refusal | robots.txt block | llms.txt |
|---|---|---|---|---|
| Cloudflare | 1570 (28.2%) | 37.8% | 14.3% | 16.6% |
| Amazon Route 53 | 1330 (23.9%) | 17.9% | 26.7% | 21.8% |
| Akamai | 276 (5%) | 30.4% | 20.5% | 17.4% |
| 170 (3.1%) | 9.4% | 19.7% | 19.4% | |
| NS1 (IBM) | 104 (1.9%) | 26% | 27.3% | 20.2% |
| Microsoft Azure | 82 (1.5%) | 15.9% | 21.3% | 2.4% |
| UltraDNS (Vercara) | 63 (1.1%) | 31.7% | 18.6% | 15.9% |
| Tencent Cloud (DNSPod) | 54 (1%) | 11.1% | 17.2% | 13% |
| Alibaba Cloud | 51 (0.9%) | 11.8% | 5.4% | 5.9% |
| GoDaddy | 45 (0.8%) | 11.1% | 5.3% | 22.2% |
| RU-CENTER | 30 (0.5%) | 13.3% | 8.7% | 13.3% |
| Namecheap | 23 (0.4%) | 21.7% | 10% | 0% |
| edrive-hosting.cz | 23 (0.4%) | 0% | 100% | 0% |
| ClouDNS | 22 (0.4%) | 22.7% | 11.8% | 4.5% |
| DNS Made Easy | 21 (0.4%) | 28.6% | 40% | 19% |
| Yandex | 18 (0.3%) | 0% | 0% | 0% |
| selectel.ru | 18 (0.3%) | 16.7% | 7.7% | 5.6% |
| nic.in | 17 (0.3%) | 23.5% | 0% | 0% |
| alibabadns.com | 14 (0.3%) | 0% | 9.1% | 28.6% |
| Gandi | 13 (0.2%) | 23.1% | 30% | 15.4% |
| CSC | 11 (0.2%) | 27.3% | 12.5% | 9.1% |
| DNSimple | 11 (0.2%) | 9.1% | 11.1% | 27.3% |
| centralnic.net | 11 (0.2%) | 0% | 0% | 0% |
| d-53.info | 11 (0.2%) | 27.3% | 33.3% | 9.1% |
| IONOS | 10 (0.2%) | 0% | 0% | 10% |
| arvancdn.ir | 10 (0.2%) | 0% | 0% | 10% |
| constellix.com | 10 (0.2%) | 10% | 28.6% | 0% |
| Other (499 providers, each under 10 sites) | 795 (14.3%) | 17.6% | 20.5% | 9.8% |
| Own nameservers | 726 (13.1%) | 13.4% | 10.8% | 11.8% |
| Unidentified | 20 (0.4%) | 15% | 0% | 5% |
Network
who holds the address the site answers from (A record, origin autonomous system). 32 named.
| Network | Sites | Edge refusal | robots.txt block | llms.txt |
|---|---|---|---|---|
| Cloudflare | 1545 (27.8%) | 37.2% | 14.1% | 19.2% |
| Amazon | 1241 (22.3%) | 17.9% | 25.3% | 20.5% |
| Fastly | 267 (4.8%) | 21.3% | 35.1% | 12.4% |
| 239 (4.3%) | 13.4% | 23.6% | 15.9% | |
| Akamai | 220 (4%) | 38.6% | 24.4% | 11.8% |
| Microsoft | 114 (2.1%) | 11.4% | 11.6% | 8.8% |
| Automattic | 72 (1.3%) | 37.5% | 42.9% | 19.4% |
| Alibaba Cloud | 70 (1.3%) | 11.4% | 4.7% | 12.9% |
| OVHcloud | 68 (1.2%) | 16.2% | 14.3% | 5.9% |
| Hetzner | 50 (0.9%) | 16% | 7.9% | 6% |
| Imperva | 44 (0.8%) | 9.1% | 14.3% | 15.9% |
| China Telecom | 33 (0.6%) | 21.2% | 5.9% | 3% |
| HLL | 31 (0.6%) | 9.7% | 7.4% | 22.6% |
| JSC Selectel | 30 (0.5%) | 13.3% | 16.7% | 10% |
| DDOS-Guard | 25 (0.4%) | 24% | 4.5% | 24% |
| Tencent Cloud | 19 (0.3%) | 10.5% | 42.9% | 15.8% |
| Yandex | 19 (0.3%) | 5.3% | 0% | 0% |
| DataWeb Global Group | 17 (0.3%) | 17.6% | 14.3% | 0% |
| VK | 17 (0.3%) | 0% | 0% | 5.9% |
| DigitalOcean | 16 (0.3%) | 12.5% | 9.1% | 6.3% |
| National Informatics Centre | 16 (0.3%) | 25% | 0% | 0% |
| CDN77 | 13 (0.2%) | 15.4% | 0% | 23.1% |
| Huawei Cloud | 13 (0.2%) | 7.7% | 0% | 7.7% |
| IQWeb FZ- | 12 (0.2%) | 8.3% | 8.3% | 0% |
| Oracle | 12 (0.2%) | 33.3% | 50% | 0% |
| Artcompanycz | 11 (0.2%) | 0% | 100% | 0% |
| Internet Initiative Japan | 11 (0.2%) | 18.2% | 22.2% | 9.1% |
| Ionos | 11 (0.2%) | 0% | 0% | 18.2% |
| Servicepipe | 11 (0.2%) | 36.4% | 0% | 9.1% |
| Verein zur Foerderung eines Deutschen Forschungsnetzes e.V | 11 (0.2%) | 0% | 9.1% | 0% |
| China Unicom | 10 (0.2%) | 10% | 50% | 0% |
| F5 Networks | 10 (0.2%) | 10% | 0% | 0% |
| Other (834 providers, each under 10 sites) | 1275 (22.9%) | 15.2% | 14.4% | 11.5% |
| Unidentified | 6 (0.1%) | 50% | 0% | 16.7% |
Bot protection
a bot-management product detected on the site. 5 named.
| Bot protection | Sites | Edge refusal | robots.txt block | llms.txt |
|---|---|---|---|---|
| Cloudflare Bot Management | 573 (10.3%) | 26.5% | 17.7% | 27.2% |
| Akamai Bot Manager | 99 (1.8%) | 52.5% | 11.6% | 20.2% |
| Imperva | 51 (0.9%) | 7.8% | 15% | 17.6% |
| DataDome | 12 (0.2%) | 41.7% | 63.6% | 8.3% |
| HUMAN (PerimeterX) | 11 (0.2%) | 45.5% | 45.5% | 18.2% |
| Other (1 providers, each under 10 sites) | 7 (0.1%) | 57.1% | 16.7% | 14.3% |
| None detected | 4724 (85%) | 22.2% | 18.7% | 14.4% |
| Not read | 82 (1.5%) | 18.3% | 30.6% | 7.3% |
Site platform
the CMS, store builder or hosting product the site runs on. 13 named.
| Site platform | Sites | Edge refusal | robots.txt block | llms.txt |
|---|---|---|---|---|
| WordPress | 734 (13.2%) | 32.2% | 16.3% | 21.5% |
| Google Cloud | 179 (3.2%) | 14.5% | 26.4% | 15.6% |
| Drupal | 167 (3%) | 25.7% | 6.2% | 7.8% |
| Vercel | 127 (2.3%) | 6.3% | 9.6% | 56.7% |
| Webflow | 113 (2%) | 13.3% | 11% | 34.5% |
| Microsoft Azure | 64 (1.2%) | 12.5% | 7.5% | 7.8% |
| AWS Elastic Load Balancing | 54 (1%) | 20.4% | 17% | 14.8% |
| Netlify | 53 (1%) | 3.8% | 4.3% | 34% |
| HubSpot CMS | 36 (0.6%) | 2.8% | 9.4% | 22.2% |
| Framer | 23 (0.4%) | 13% | 9.5% | 34.8% |
| GitHub Pages | 20 (0.4%) | 5% | 11.1% | 20% |
| Shopify | 18 (0.3%) | 5.6% | 6.3% | 66.7% |
| Wix | 12 (0.2%) | 8.3% | 9.1% | 91.7% |
| Other (9 providers, each under 10 sites) | 26 (0.5%) | 23.1% | 0% | 11.5% |
| Not identified | 3851 (69.3%) | 23.6% | 20.7% | 12.5% |
| Not read | 82 (1.5%) | 18.3% | 30.6% | 7.3% |
Where the addresses are registered
Each dot is a country, sized by the mapped sites whose address block is registered there and shaded by the metric chosen above. Drag to turn the globe; tap a dot for its numbers. the registration country of each site's IPv4 address block (Team Cymru origin record), not where the server sits. 50 countries shown; 22 with fewer than 3 sites pooled (31 sites); 13 sites with a regional or unknown registration not placed.
| Country | Sites | Edge refusal | robots.txt block | llms.txt |
|---|---|---|---|---|
| US | 4108 (73.9%) | 26.1% | 20.5% | 17.7% |
| RU | 240 (4.3%) | 13.8% | 7.7% | 10% |
| DE | 146 (2.6%) | 10.3% | 13% | 8.2% |
| CN | 113 (2%) | 12.4% | 17.1% | 4.4% |
| FR | 98 (1.8%) | 17.3% | 13.9% | 6.1% |
| SG | 73 (1.3%) | 31.5% | 13.6% | 11% |
| GB | 70 (1.3%) | 15.7% | 10.2% | 25.7% |
| NL | 65 (1.2%) | 20% | 14% | 4.6% |
| CA | 46 (0.8%) | 15.2% | 10.8% | 17.4% |
| IR | 44 (0.8%) | 6.8% | 0% | 9.1% |
| CZ | 42 (0.8%) | 9.5% | 35.1% | 7.1% |
| JP | 41 (0.7%) | 19.5% | 14.7% | 9.8% |
Where edge refusals happen
Of the 1287 sites that refused an AI identity at the edge while serving the browser, the networks they answer from. Lift = the network’s refusal rate over the frame’s (23.2%).
| Network | Refusing sites | Share of all refusals | Rate | Lift |
|---|---|---|---|---|
| Cloudflare | 575 | 44.7% | 37.2% | 1.6× |
| Amazon | 222 | 17.2% | 17.9% | 0.77× |
| Other (834 providers, each under 10 sites) | 194 | 15.1% | 15.2% | 0.66× |
| Akamai | 85 | 6.6% | 38.6% | 1.66× |
| Fastly | 57 | 4.4% | 21.3% | 0.92× |
| 32 | 2.5% | 13.4% | 0.58× | |
| Automattic | 27 | 2.1% | 37.5% | 1.62× |
| Microsoft | 13 | 1% | 11.4% | 0.49× |
| OVHcloud | 11 | 0.9% | 16.2% | 0.7× |
| Alibaba Cloud | 8 | 0.6% | 11.4% | 0.49× |
Who blocks in robots.txt, by platform
| Platform | Blocking sites | Share of all blocks | Rate | Lift |
|---|---|---|---|---|
| Not identified | 634 | 74.9% | 20.7% | 1.1× |
| WordPress | 106 | 12.5% | 16.3% | 0.87× |
| Google Cloud | 39 | 4.6% | 26.4% | 1.4× |
| Vercel | 11 | 1.3% | 9.6% | 0.51× |
| Webflow | 11 | 1.3% | 11% | 0.59× |
| Drupal | 9 | 1.1% | 6.2% | 0.33× |
| AWS Elastic Load Balancing | 8 | 0.9% | 17% | 0.9× |
| Microsoft Azure | 3 | 0.4% | 7.5% | 0.4× |
| HubSpot CMS | 3 | 0.4% | 9.4% | 0.5× |
| Netlify | 2 | 0.2% | 4.3% | 0.23× |
Each reachable site in the frame is placed in four layers: its DNS host (NS records matched to the operator, or the nameservers' own domain), the holder of the network its address sits in (A record, origin autonomous system from Team Cymru's IP-to-ASN zone, holder name normalised), and the bot-protection product and site platform CrawlCheck's technology record detected on it ("Not read" where the site has no record yet; "None detected" means none of the products we detect). Each site carries what the census run saw: refused at least one AI identity (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot) at the edge while serving the browser; blocked an AI crawler in robots.txt (share of sites whose robots.txt was served); served /llms.txt; declared Content-Signal. A provider is named when at least 10 mapped sites sit behind it, smaller ones are pooled as Other; links between layers under 3 sites are left out. A share is where sites are, not what a provider does: a refusal at the edge may be the provider's default or the site's own rule. No site is named.
Signed JSON: /api/v1/agentic-web-map?month=2026-10 · one domain: /api/v1/agentic-web-map?domain=example.com · census method: /api/census.