CrawlCheck
Data · census · weekly

Agentic web census, week 2026-W44

Sealed 2026-11-02. The same frozen sets of the most-visited sites, measured on a schedule from outside: what they tell and serve AI agents this week against last week. No site is named.

top 1,000, daily

Run 31 (finished 2026-11-01), 991 sites, compared with run 24 (2026-10-25).

Biggest moves: Of sites naming an MCP endpoint, one completes the handshake or asks for credentials +3.8 pt (2 gained, 0 lost); GPTBot refused where the browser got 2xx +1.2 pt (6 gained, 0 lost); At least one AI identity is refused (non-2xx) where the browser got 2xx +0.8 pt (9 gained, 5 lost); ClaudeBot refused where the browser got 2xx +0.8 pt (7 gained, 3 lost); robots.txt closes / to Amazonbot +0.8 pt (3 gained, 0 lost).

FigureThis weekChange, same sitesSites +/−
Homepage answers CrawlCheck's census fetch (2xx after redirects)
of 991
51.4%0 pt+11 / −11
Homepage refuses CrawlCheck's census fetch (400, 401, 403 or 429): blocked or rate-limited, not down
of 991
12.3%——
No website answers at that name (5xx, 530, timeout or no answer)
of 991
32.6%——
Serves a robots.txt (200, not HTML)
of 509
78.4%-0.2 pt+4 / −5
robots.txt names a sitemap
of 399
72.2%+0.5 pt+2 / −0
robots.txt carries a Content-Signal line
of 399
4.8%+0.5 pt+2 / −0
Content-Signal says ai-train=no
of 399
3.3%+0.3 pt+1 / −0
Content-Signal says ai-input=no or search=no
of 399
1%0 pt+0 / −0
robots.txt closes / to at least one of the 12 AI agents
of 399
25.3%+0.5 pt+2 / −0
robots.txt closes / to GPTBot
of 399
16.8%+0.3 pt+1 / −0
robots.txt closes / to OAI-SearchBot
of 399
7.5%0 pt+0 / −0
robots.txt closes / to ChatGPT-User
of 399
9.8%0 pt+0 / −0
robots.txt closes / to ClaudeBot
of 399
15.5%+0.5 pt+2 / −0
robots.txt closes / to Claude-SearchBot
of 399
8.3%+0.5 pt+2 / −0
robots.txt closes / to PerplexityBot
of 399
11.5%0 pt+0 / −0
robots.txt closes / to Google-Extended
of 399
14%+0.3 pt+1 / −0
robots.txt closes / to Applebot-Extended
of 399
12.5%+0.5 pt+2 / −0
robots.txt closes / to CCBot
of 399
19.5%+0.3 pt+1 / −0
robots.txt closes / to Bytespider
of 399
19.3%+0.5 pt+2 / −0
robots.txt closes / to Meta-ExternalAgent
of 399
15%+0.5 pt+2 / −0
robots.txt closes / to Amazonbot
of 399
14%+0.8 pt+3 / −0
Serves /llms.txt
of 509
21.6%0 pt+1 / −1
Serves /agents.md
of 509
2.4%0 pt+0 / −0
Publishes an agent-skills index
of 509
2.6%+0.2 pt+1 / −0
Publishes an MCP server card
of 509
2.6%0 pt+0 / −0
Publishes an API catalog (RFC 9727)
of 509
2.4%+0.2 pt+1 / −0
Publishes an A2A agent card
of 509
0.6%0 pt+0 / −0
Declares any agent capability (any of the four above)
of 509
3.3%+0.4 pt+2 / −0
At least one AI identity is refused (non-2xx) where the browser got 2xx
of 509
18.9%+0.8 pt+9 / −5
At least one AI identity gets under 70% of the browser's bytes (both 2xx)
of 509
3.9%+0.2 pt+2 / −1
Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)
of 509
10.2%-0.4 pt+0 / −2
An MCP endpoint completes the handshake (initialize and tools/list; no tool called)
of 509
1.6%0 pt+0 / −0
Of sites naming an MCP endpoint, one completes the handshake or asks for credentials
of 52
80.8%+3.8 pt+2 / −0
The A2A agent card names an endpoint url
of 509
0.4%0 pt+0 / −0
Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)
of 2
50%0 pt+0 / −0
Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)
of 509
0.2%0 pt+0 / −0
GPTBot refused where the browser got 2xx
of 509
16.5%+1.2 pt+6 / −0
ClaudeBot refused where the browser got 2xx
of 509
14.9%+0.8 pt+7 / −3
OAI-SearchBot refused where the browser got 2xx
of 509
10.8%+0.2 pt+3 / −2
PerplexityBot refused where the browser got 2xx
of 509
11.8%+0.4 pt+5 / −3

top 10,000, daily

Run 5 (finished 2026-10-31), 9776 sites, compared with run 4 (2026-10-24).

Biggest moves: robots.txt closes / to Bytespider +2 pt (89 gained, 1 lost); robots.txt closes / to CCBot +2 pt (88 gained, 1 lost); robots.txt closes / to Amazonbot +2 pt (87 gained, 2 lost); robots.txt closes / to Google-Extended +1.9 pt (86 gained, 4 lost); robots.txt closes / to at least one of the 12 AI agents +1.8 pt (81 gained, 1 lost).

FigureThis weekChange, same sitesSites +/−
Homepage answers CrawlCheck's census fetch (2xx after redirects)
of 9776
57.5%+0.7 pt+228 / −160
Homepage refuses CrawlCheck's census fetch (400, 401, 403 or 429): blocked or rate-limited, not down
of 9776
13.7%——
No website answers at that name (5xx, 530, timeout or no answer)
of 9776
26.8%——
Serves a robots.txt (200, not HTML)
of 5621
81.7%+0.5 pt+63 / −36
robots.txt names a sitemap
of 4594
70.1%+0.1 pt+15 / −10
robots.txt carries a Content-Signal line
of 4594
6.2%+1.7 pt+75 / −0
Content-Signal says ai-train=no
of 4594
4.8%+1.7 pt+74 / −0
Content-Signal says ai-input=no or search=no
of 4594
0.5%0 pt+0 / −0
robots.txt closes / to at least one of the 12 AI agents
of 4594
21.9%+1.8 pt+81 / −1
robots.txt closes / to GPTBot
of 4594
16.2%+1.6 pt+72 / −1
robots.txt closes / to OAI-SearchBot
of 4594
6.2%0 pt+4 / −3
robots.txt closes / to ChatGPT-User
of 4594
8.1%0 pt+5 / −3
robots.txt closes / to ClaudeBot
of 4594
14.9%+1.8 pt+78 / −1
robots.txt closes / to Claude-SearchBot
of 4594
5.7%+0.1 pt+6 / −2
robots.txt closes / to PerplexityBot
of 4594
9.3%+0.3 pt+17 / −3
robots.txt closes / to Google-Extended
of 4594
13.4%+1.9 pt+86 / −4
robots.txt closes / to Applebot-Extended
of 4594
12.9%+1.8 pt+78 / −1
robots.txt closes / to CCBot
of 4594
17.2%+2 pt+88 / −1
robots.txt closes / to Bytespider
of 4594
16.9%+2 pt+89 / −1
robots.txt closes / to Meta-ExternalAgent
of 4594
13.7%+1.8 pt+78 / −1
robots.txt closes / to Amazonbot
of 4594
13.1%+2 pt+87 / −2
Serves /llms.txt
of 5621
16.6%+0.3 pt+18 / −4
Serves /agents.md
of 5621
1.3%0 pt+0 / −0
Publishes an agent-skills index
of 5621
1%0 pt+2 / −1
Publishes an MCP server card
of 5621
1%0 pt+1 / −1
Publishes an API catalog (RFC 9727)
of 5621
1.1%0 pt+2 / −3
Publishes an A2A agent card
of 5621
0.2%0 pt+0 / −2
Declares any agent capability (any of the four above)
of 5621
1.4%0 pt+3 / −3
At least one AI identity is refused (non-2xx) where the browser got 2xx
of 5621
23.7%+0.3 pt+82 / −67
At least one AI identity gets under 70% of the browser's bytes (both 2xx)
of 5621
2.5%+0.1 pt+12 / −9
Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)
of 5621
5.6%0 pt+9 / −7
An MCP endpoint completes the handshake (initialize and tools/list; no tool called)
of 5621
0.9%-0.1 pt+1 / −4
Of sites naming an MCP endpoint, one completes the handshake or asks for credentials
of 315
79.4%+1 pt+3 / −0
The A2A agent card names an endpoint url
of 5621
0.1%0 pt+0 / −1
Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)
of 8
50%0 pt+0 / −0
Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)
of 5621
0.1%0 pt+0 / −0
GPTBot refused where the browser got 2xx
of 5621
19.7%-0.4 pt+37 / −57
ClaudeBot refused where the browser got 2xx
of 5621
20.3%+0.1 pt+58 / −55
OAI-SearchBot refused where the browser got 2xx
of 5621
13.3%+0.3 pt+46 / −29
PerplexityBot refused where the browser got 2xx
of 5621
13.4%-0.1 pt+27 / −32

Each frame compares the last run that finished in this ISO week (UTC) with the last run that finished before it. Change is like-for-like: points = (sites gained - sites lost) / sites compared, counting only sites measured in both runs and inside the figure's denominator both times, so a site that did not answer in one run cannot move a figure. points_all is the plain difference of the two percentages, and denominator_entered / denominator_left count the sites that moved in or out of a figure's base (often a site that did not answer one run). No site is named; per-site records: https://crawlcheck.io/api/census/records.

Signed JSON: /api/v1/census/weekly?week=2026-W44 · feed: Atom · method: /api/census. Editions: 2026-W46 · 2026-W45 · 2026-W44 · 2026-W43 · 2026-W42 · 2026-W41 · 2026-W40.

Check it yourself

Every number here comes from the signed JSON at /api/v1/census/weekly; each copy carries its own sha256 and an Ed25519 signature by key 1hHmE3fo1rg5…. Verify it with no call to us:

node crawlcheck-verify.mjs --online census-weekly.json

The verifier: /crawlcheck-verify.mjs, zero dependencies.

Editions

One sealed edition per ISO week, never edited. Feed: Atom · all instruments: /api/v1/instruments.