CrawlCheck
Data · census · weekly

Agentic web census, week 2026-W42

Sealed 2026-10-20. The same frozen sets of the most-visited sites, measured on a schedule from outside: what they tell and serve AI agents this week against last week. No site is named.

top 1,000, daily

Run 17 (finished 2026-10-18), 991 sites, compared with run 10 (2026-10-11).

Biggest moves: Homepage answers a browser (2xx after redirects) +2.3 pt (40 gained, 17 lost); PerplexityBot refused where the browser got 2xx +1 pt (5 gained, 0 lost); ClaudeBot refused where the browser got 2xx +0.8 pt (8 gained, 4 lost); Serves /llms.txt +0.8 pt (5 gained, 1 lost); At least one AI identity is refused (non-2xx) where the browser got 2xx +0.6 pt (7 gained, 4 lost).

FigureThis weekChange, same sitesSites +/−
Homepage answers a browser (2xx after redirects)
of 991
54.3%+2.3 pt+40 / −17
Serves a robots.txt (200, not HTML)
of 538
77.7%-0.2 pt+0 / −1
robots.txt names a sitemap
of 418
70.8%+0.5 pt+2 / −0
robots.txt carries a Content-Signal line
of 418
3.8%+0.3 pt+1 / −0
Content-Signal says ai-train=no
of 418
2.4%+0.3 pt+1 / −0
Content-Signal says ai-input=no or search=no
of 418
1%0 pt+0 / −0
robots.txt closes / to at least one of the 12 AI agents
of 418
23.2%+0.3 pt+1 / −0
robots.txt closes / to GPTBot
of 418
14.8%+0.3 pt+1 / −0
robots.txt closes / to OAI-SearchBot
of 418
7.2%0 pt+0 / −0
robots.txt closes / to ChatGPT-User
of 418
9.3%0 pt+0 / −0
robots.txt closes / to ClaudeBot
of 418
14.6%+0.3 pt+1 / −0
robots.txt closes / to Claude-SearchBot
of 418
7.7%0 pt+0 / −0
robots.txt closes / to PerplexityBot
of 418
11.2%0 pt+0 / −0
robots.txt closes / to Google-Extended
of 418
12.2%+0.3 pt+2 / −1
robots.txt closes / to Applebot-Extended
of 418
11.2%0 pt+0 / −0
robots.txt closes / to CCBot
of 418
18.4%+0.3 pt+1 / −0
robots.txt closes / to Bytespider
of 418
17.9%+0.3 pt+1 / −0
robots.txt closes / to Meta-ExternalAgent
of 418
14.1%0 pt+0 / −0
robots.txt closes / to Amazonbot
of 418
13.2%+0.3 pt+1 / −0
Serves /llms.txt
of 538
21.2%+0.8 pt+5 / −1
Serves /agents.md
of 538
2.6%0 pt+0 / −0
Publishes an agent-skills index
of 538
2.4%+0.2 pt+1 / −0
Publishes an MCP server card
of 538
2.4%+0.2 pt+1 / −0
Publishes an API catalog (RFC 9727)
of 538
2.2%+0.2 pt+1 / −0
Publishes an A2A agent card
of 538
0.6%+0.2 pt+1 / −0
Declares any agent capability (any of the four above)
of 538
3%+0.2 pt+1 / −0
At least one AI identity is refused (non-2xx) where the browser got 2xx
of 538
22.3%+0.6 pt+7 / −4
At least one AI identity gets under 70% of the browser's bytes (both 2xx)
of 538
4.5%+0.4 pt+3 / −1
Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)
of 538
10.2%+0.4 pt+3 / −1
An MCP endpoint completes the handshake (initialize and tools/list; no tool called)
of 538
1.7%+0.2 pt+1 / −0
Of sites naming an MCP endpoint, one completes the handshake or asks for credentials
of 55
72.7%0 pt+0 / −0
The A2A agent card names an endpoint url
of 538
0.2%0 pt+0 / −0
Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)
of 1
0%0 pt+0 / −0
Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)
of 538
0.2%0 pt+0 / −0
GPTBot refused where the browser got 2xx
of 538
19.1%0 pt+2 / −2
ClaudeBot refused where the browser got 2xx
of 538
18.6%+0.8 pt+8 / −4
OAI-SearchBot refused where the browser got 2xx
of 538
13.9%-0.2 pt+1 / −2
PerplexityBot refused where the browser got 2xx
of 538
15.2%+1 pt+5 / −0

top 10,000, weekly

Run 3 (finished 2026-10-17), 9776 sites, compared with run 2 (2026-10-10).

Biggest moves: Homepage answers a browser (2xx after redirects) +2.4 pt (425 gained, 195 lost); robots.txt closes / to Amazonbot +0.4 pt (22 gained, 5 lost); robots.txt closes / to Applebot-Extended +0.4 pt (21 gained, 5 lost); robots.txt closes / to Meta-ExternalAgent +0.4 pt (21 gained, 4 lost); Of sites naming an MCP endpoint, one completes the handshake or asks for credentials +0.4 pt (1 gained, 0 lost).

FigureThis weekChange, same sitesSites +/−
Homepage answers a browser (2xx after redirects)
of 9776
59.2%+2.4 pt+425 / −195
Serves a robots.txt (200, not HTML)
of 5789
79.3%-0.1 pt+34 / −37
robots.txt names a sitemap
of 4588
69.8%+0.2 pt+13 / −3
robots.txt carries a Content-Signal line
of 4588
3.5%+0.2 pt+9 / −2
Content-Signal says ai-train=no
of 4588
2.2%+0.1 pt+8 / −2
Content-Signal says ai-input=no or search=no
of 4588
0.5%0 pt+0 / −0
robots.txt closes / to at least one of the 12 AI agents
of 4588
19.6%+0.2 pt+14 / −5
robots.txt closes / to GPTBot
of 4588
13.8%+0.1 pt+10 / −5
robots.txt closes / to OAI-SearchBot
of 4588
6.4%0 pt+6 / −4
robots.txt closes / to ChatGPT-User
of 4588
8.1%-0.2 pt+4 / −11
robots.txt closes / to ClaudeBot
of 4588
12.5%+0.1 pt+10 / −5
robots.txt closes / to Claude-SearchBot
of 4588
5.9%+0.1 pt+5 / −1
robots.txt closes / to PerplexityBot
of 4588
9.2%0 pt+6 / −4
robots.txt closes / to Google-Extended
of 4588
10.8%+0.3 pt+20 / −6
robots.txt closes / to Applebot-Extended
of 4588
10.4%+0.4 pt+21 / −5
robots.txt closes / to CCBot
of 4588
14.6%+0.2 pt+11 / −4
robots.txt closes / to Bytespider
of 4588
14.2%+0.2 pt+11 / −4
robots.txt closes / to Meta-ExternalAgent
of 4588
11.2%+0.4 pt+21 / −4
robots.txt closes / to Amazonbot
of 4588
10.7%+0.4 pt+22 / −5
Serves /llms.txt
of 5789
15.6%+0.2 pt+21 / −8
Serves /agents.md
of 5789
1.3%0 pt+4 / −3
Publishes an agent-skills index
of 5789
0.9%0 pt+1 / −1
Publishes an MCP server card
of 5789
0.9%0 pt+2 / −1
Publishes an API catalog (RFC 9727)
of 5789
1.1%0 pt+1 / −1
Publishes an A2A agent card
of 5789
0.3%0 pt+0 / −2
Declares any agent capability (any of the four above)
of 5789
1.3%0 pt+3 / −1
At least one AI identity is refused (non-2xx) where the browser got 2xx
of 5789
25.3%+0.1 pt+73 / −70
At least one AI identity gets under 70% of the browser's bytes (both 2xx)
of 5789
2.5%0 pt+14 / −15
Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)
of 5789
5.3%+0.1 pt+9 / −3
An MCP endpoint completes the handshake (initialize and tools/list; no tool called)
of 5789
0.9%0 pt+2 / −0
Of sites naming an MCP endpoint, one completes the handshake or asks for credentials
of 305
77.7%+0.4 pt+1 / −0
The A2A agent card names an endpoint url
of 5789
0.1%0 pt+0 / −0
Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)
of 8
37.5%0 pt+0 / −0
Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)
of 5789
0.1%0 pt+0 / −0
GPTBot refused where the browser got 2xx
of 5789
21.7%0 pt+39 / −39
ClaudeBot refused where the browser got 2xx
of 5789
22.2%+0.2 pt+54 / −45
OAI-SearchBot refused where the browser got 2xx
of 5789
14.9%-0.1 pt+33 / −37
PerplexityBot refused where the browser got 2xx
of 5789
15.4%-0.1 pt+38 / −41

Each frame compares the last run that finished in this ISO week (UTC) with the last run that finished before it. Change is like-for-like: points = (sites gained - sites lost) / sites compared, counting only sites measured in both runs and inside the figure's denominator both times, so a site that did not answer in one run cannot move a figure. points_all is the plain difference of the two percentages, and denominator_entered / denominator_left count the sites that moved in or out of a figure's base (often a site that did not answer one run). No site is named; per-site records: https://crawlcheck.io/api/census/records.

Signed JSON: /api/v1/census/weekly?week=2026-W42 · feed: Atom · method: /api/census. Editions: 2026-W46 · 2026-W45 · 2026-W44 · 2026-W43 · 2026-W42 · 2026-W41 · 2026-W40.

Check it yourself

Every number here comes from the signed JSON at /api/v1/census/weekly; each copy carries its own sha256 and an Ed25519 signature by key 1hHmE3fo1rg5…. Verify it with no call to us:

node crawlcheck-verify.mjs --online census-weekly.json

The verifier: /crawlcheck-verify.mjs, zero dependencies.

Editions

One sealed edition per ISO week, never edited. Feed: Atom · all instruments: /api/v1/instruments.