Agent view · Findings · 2026-09-23

Sites measured is not people: what our own logs showed

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/our-homepage-says-6557-sites-measured-six-people-ran-a-scan-this-week: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

Findings · 2026-09-23 · By [VSNARY | Emmanuel Orta](/author)

# Sites measured is not people: what our own logs showed

The sites-measured figure on our homepage is true, and it is not demand. One week of our own request logs shows who visits a site about crawlers: mostly crawlers.

The sites-measured figure on CrawlCheck's homepage counts distinct domains with a measured record, most of them scanned to build the benchmark population that every finding rate on the site rests on. It is not a count of people. In one week of the site's own request logs, scans from anyone other than the operator came to about six, from five places, while the busiest page, pricing, was fetched thousands of times by SEO and AI crawlers. The figure stays, labelled as what it is, next to a count of visitor scans.

On this page

- Who fetched the site in one week?
- How do you tell a crawler from a reader in your own logs?
- How many people ran a scan?
- What changed?

The number on the homepage is counted to completion: the distinct domains that carry a measured record. What it is not is people. Most of those domains were measured on purpose, so that every finding rate on [the dataset page](/data) has a population under it. A rate means nothing without enough sites behind it, and that is what the figure is for. Under a hero headline, though, it reads as demand. This note corrects that reading.

**Live, as you read this:** the corpus now holds 503,711 domains across 6,399 scans. The figures in this piece were measured on the date above; this line is not.

## Who fetched the site in one week? #

We read seven days of the site's own request logs, page views only. The site that measures crawlers turns out to be visited mostly by them.

| Path | Requests, 7 days | Who |
| --- | --- | --- |
| /pricing | 9,485 | SEO and AI crawlers, almost entirely |
| / (homepage) | 2,608 | mixed; the only page with a large human share |
| /demo | 425 | the demo report |
| each blog post | 45–60 | nearly identical on every post |

The pricing page was the busiest page on the site by a factor of three, and almost no human saw it. Every report and registry page links to it with the scanned domain in the query string, so crawlers walking those pages follow thousands of variants of one page. The variants carry a [canonical](/glossary/canonical) to the bare [URL](/glossary/url) and are served from cache, so the cost is close to zero and it is not a defect. It is just not demand.

## How do you tell a crawler from a reader in your own logs? #

By shape. A post people read has a spike, a tail, and a count that differs from its neighbours. Forty-five to sixty requests on every post, regardless of age or subject, is the signature of sitemap walkers. If your analytics show every page with roughly the same count, you are looking at crawlers. Before counting any of them as a named crawler, check them: [claimed hits are not verified visits](/blog/how-much-ai-crawler-traffic-is-forged), and [a missing referrer is not direct traffic](/blog/ninety-three-percent-of-our-arrivals-sent-no-referrer) either.

## How many people ran a scan? #

| Scans, 7 days | Count |
| --- | --- |
| Successful scan requests | 557 |
| From our own browsers and tests | about 545 |
| From anyone else | about 6, from five places |

That is the honest denominator for anything this site says about itself, and it is the same discipline CrawlCheck applies to everyone else: a count that mixes the operator's own traffic with the public's is not a count of the public.

## What changed? #

The sites-measured figure stays, because it is true and because the benchmark population is the reason the rates on this site mean anything. It gained a neighbour: scans by visitors in the last thirty days, with our own traffic excluded. When that number is small, it says so. It was not the first figure here that was correct and misleading at once; the first was [a cohort percentage that only means something with a second day under it](/blog/the-two-facts-almost-nobody-publishes). The fix each time is the same: name the population, or do not print the number.

This is one of the corrections CrawlCheck publishes when its own output was wrong. [All of them are listed together](/blog?kind=corrections).

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-homepage-says-6557-sites-measured-six-people-ran-a-scan-this-week%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-homepage-says-6557-sites-measured-six-people-ran-a-scan-this-week%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Sites%20measured%20is%20not%20people%3A%20what%20our%20own%20logs%20showed)[Email](mailto:?subject=Sites%20measured%20is%20not%20people%3A%20what%20our%20own%20logs%20showed&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-homepage-says-6557-sites-measured-six-people-ran-a-scan-this-week)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Is the sites-measured figure on the CrawlCheck homepage wrong?

No. It counts distinct domains that carry a measured record, counted to completion. Most were measured to build the benchmark population behind the finding rates, not scanned by visitors.

### How many people actually used CrawlCheck that week?

About six scans came from addresses that were not the operator's, from five places. Nearly all other successful scan requests were the operator's own browsers and tests.

### Why does a pricing page get so much crawler traffic?

When every report links to it with a different query string, crawlers follow each variant as a separate URL. A canonical to the bare URL and caching keep the cost near zero, but the requests are not demand.

### How do you tell a bot from a reader on a blog?

Shape. A post people read has a spike and a tail and a count that differs from its neighbours. Near-identical counts on every post regardless of age point to sitemap walkers.

## Related findings

[2026-08-28 Is Cloudflare blocking AI crawlers on your site? Since 15 September 2026 Cloudflare blocks Training crawlers by default, which can take Googlebot and Bingbot with them. How to che](/guides/is-cloudflare-blocking-ai-crawlers-on-your-site)[2026-09-23 Correction: the owner's own scans came back locked The site owner's scans came back locked while the report page opened in full. What was wrong, what changed, and how to test gated](/blog/the-owner-could-not-read-his-own-scans)[2026-09-30 Our extension read a page as 36.5% JavaScript. It was zero. Our extension called a server-rendered page 36.5% JavaScript; the true figure was zero. Two causes, both inflating, and what versi](/blog/the-page-our-extension-read-as-36-percent-javascript-was-zero)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit, and placement is never for sale.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [Entity](/registry/entity) · [AI policy](/registry/aipolicy) · [graphicriver.net](/registry/graphicriver.net) · [lhh.com](/registry/lhh.com) · [nomic.ai](/registry/nomic.ai) · [regalgmc.com](/registry/regalgmc.com) · [starling-ford.com](/registry/starling-ford.com) · [xata.io](/registry/xata.io) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) | **Certificates:** [applovin.com](/certificate/applovin.com) · [harvard.edu](/certificate/harvard.edu)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Report an error](/report-error)

Same body as a file: /blog/our-homepage-says-6557-sites-measured-six-people-ran-a-scan-this-week.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.