CrawlCheck
Library · contributed findings

Submit the finding, the data and the method.

In one sentence

Submit a finding is the form for sending CrawlCheck a study you ran yourself on what AI crawlers, agents or search engines are served by websites; a person reads it, we re-run what we can from our own network, and an accepted finding is published in the library under your name with a link to your data, while a declined one comes back with the reason.

The CrawlCheck library publishes what AI crawlers, agents and search engines are actually served by websites: measured, dated, with the method shown. If you ran a study like that, send it. Accepted findings carry your name and a link to your study.

23findings in the library measured only here
~1 weekto a reply, by email
$0to submit or be published; placement is never for sale

Send a finding See what gets published →

What happens to a submission

  1. 1SendThe finding, a link to the data and the method, through this form.
  2. 2We read itEvery submission is read by a person. You get a reply by email, usually within a week.
  3. 3We re-measureWhat can be re-run from our own network is re-run. What cannot be checked is said to be unchecked.
  4. 4Published or declinedAccepted: written up under your name with a link to your study. Declined: you get the reason.

What we accept

  • Your own measurement. Data you collected, not a summary of someone else’s.
  • A number and a method. How many sites, requests or tools; what was fetched or counted; from where and when.
  • Something new, or something at a new size. The bar for our exclusive findings: never published, or never observed at this scale.
  • Checkable. We re-run what we can from our own network before publishing.

What we decline

  • Marketing. A product announcement written as a study.
  • Claims without data. Rankings, scores or “we found” with no dataset a reader can open.
  • Paid placement. Publication is never for sale, and nothing changes if you are a customer.
  • Naming a private business for a defect before that business has been told about it.
  • What we cannot verify or reproduce. Declined with the reason, so you can resubmit.

What a strong submission looks like

An invented example, to show the level of detail. It is not a real study.

Headline
4 in 10 recipe sites serve GPTBot a page with no ingredients list
Data
A public CSV of every URL, status code, byte count and a hash of each response
Sample
2,000 recipe pages from 400 sites, picked from one public top-sites list
Measured
3–5 October 2026, one request per identity per page, from one cloud region
Method
Each page fetched as a desktop browser and as GPTBot with its published user agent; “no ingredients list” means no list markup and none of the browser copy’s ingredient lines; sites behind a login or bot challenge were left out and counted

Send a finding

Required fields are marked. Nothing you send is published until you hear from us.

1 The finding

One sentence a reader could quote. Put the number in it.

Where a reader can see the data. This is the page we credit and link.

The result in a few sentences, with the numbers.

0 of at least 80 characters

2 How it was measured

What was fetched or counted, which user agent or client, from where, when, and anything excluded.

0 of at least 40 characters

3 About you

Only to reply. Never published.

We print this next to the finding. A tie does not disqualify a study; an undisclosed one does.

Recently published in the library

2026-11-02 · Research · Exclusive6 min

Google's Googlebot check passes 198 of 200 rented servers

Google's verification rule accepts reverse DNS under googleusercontent.com, the domain every Google Cloud customer machine gets. In a 200-address sample, 198 passed, and two open-source verifiers accepted a rented server as Googlebot.

Read the measurements →

2026-11-02 · Research · Exclusive7 min

State of AI visibility, October 2026: 139,788 homepages

The open dataset's first full census: how many sites block AI crawlers, how many publish llms.txt, how little of a homepage is text, and what one audit cannot see.

ROBOTS_RULES_SHADOWEDSTALE_CACHE_SERVEDNO_LLMS_TXT+2

Read the measurements →

2026-11-02 · Research · Exclusive3 min

Only 6% of websites block an AI crawler: the full breakdown

Which AI crawlers 139,788 sites disallow in robots.txt, how training and search agents are treated differently, and how blocking varies by kind of site.

Read the measurements →

All 23 exclusive findings →

Questions about this page

QDo you pay for findings, or charge to publish them?
Neither. Submitting is free, publication is never for sale, and being a customer changes nothing about whether a finding is accepted.
QWho owns the data?
You do. We link to your study and dataset rather than copying it, and the write-up credits you by the name you give.
QWill you publish my finding exactly as I wrote it?
No. Accepted findings are written up in the library's format: the result, the sample, the method and its limits, plus a note of anything we re-measured and whether our numbers matched yours. You see it before it goes live.
QCan a finding name companies?
Yes, when the companies are public vendors or operators measured on their own sites. A private business found with a defect is not named until it has been told.
QWhat if I work for a company the finding is about?
Say so in the disclosure field. A tie does not disqualify a study; it is printed next to the finding. An undisclosed tie does disqualify it.