CrawlCheck

Findings · 2026-08-20 · By

AI visibility scores measure an API, not ChatGPT

The answers these tools record are real. The label on top — “your visibility in AI search” — describes something that was never measured.

They measure something real: what a model API returned for a chosen prompt list on a given day. What they usually claim is different — your visibility in ChatGPT or AI Overviews. Three substitutions sit between the two. An API is not the consumer app, which adds retrieval, tools, personalisation and memory. A hand-picked prompt list is not a property of your brand. And one answer is one draw from a distribution that changes between runs. The number is not fake; the sentence wrapped around it is.

An AI visibility tool asks the Gemini API who is the best tree service in Denver, records the answer, counts how often your brand appears and reports a percentage. Every step is real; the model genuinely produced those words. What is not real is the headline wrapped around the number: “your visibility in AI search.”

What sits between the API answer and the claim? #

What was measured:an API answer,chosen prompts, one dayWhat the label says:your visibilityin ChatGPT1 · surface: API, not the app2 · sample: a prompt list, not your brand3 · moment: one draw, not a stable shareEach substitution is a step the method never took.
The number is real. The sentence wrapped around it crosses three gaps.
What was measuredWhat the label saysWhy they differ
The Gemini or OpenAI APIyour visibility in ChatGPT or AI OverviewsChatGPT adds retrieval, tools, personalisation and memory the raw API does not reproduce; AI Overviews is a different system again, inside a results page
A hand-picked prompt lista property of your brandnobody holds the distribution of what buyers actually ask; change the list and the number changes
One answer, on one daya stable share of voicethe same prompt gives different answers between runs, and larger differences after a model update

None of this is fraud. There is no public API for the consumer surfaces, so every vendor measures what it can reach and labels it as what you asked about. The gap is in the rename, and the rename is easy to spot: it always uses a word the method never did.

How do you tell a measurement from a label? #

Ask what would have to be true for the number to be wrong, and how you would find out. If there is no answer, it is not a measurement. What survives that test is narrow: name the model, the mode (grounded or from weights) and the date, report presence or absence rather than a rank, and publish the churn between dated runs.

Ask the vendorA straight answer sounds like
Which surface?“the Gemini API”, not “AI search”
Which model, and which mode?a version string, and grounded or from weights
Which date?a date, not “recently”
What was the sample?the prompt list, or the URL and the client identity
Can I re-run it?the method, in enough detail to disagree with

The first four are facts a vendor either has or does not. The fifth decides whether you bought a measurement or a label.

Is the API number useless, then? #

No. Run against a fixed prompt list, with the model and mode pinned, on a schedule, it is a reasonable change detector. The absolute percentage means little; a series can tell you something moved, and you can then ask whether the prompt list, the model version or the mode changed. That is a useful signal and a much smaller claim than the one usually printed above it. Measuring a claim properly would need the consumer surface itself, the documents it retrieved for that user at that moment, and attribution for each sentence of the answer. Nobody outside the providers holds any of the three.

What can be measured directly? #

The retrieval side: not whether an assistant named you, but whether it could have. That is a fetch, and a fetch can be repeated. Three yes-or-no questions with a number behind each: reach (does a response arrive for a non-browser client), read (is there readable text in the bytes before anything runs) and quote (is there a self-contained passage worth lifting). CrawlCheck answers them by fetching the page as each named AI crawler and comparing the replies. The result is a status code, a ratio and a diff — evidence the input was available, never evidence of citation. How to measure AI visibility yourself walks through the same checks by hand.

Where we caught the same error in our own markup #

Our directory publishes a measured score for each listed vendor site. For a while its structured data emitted each score as a schema.org Review with a reviewRating. That was this exact substitution: a review says a person formed an opinion, and nobody did — a scanner computed a number. It now emits an Observation, with the score as measured values and a link to the evidence report it came from. The rule we took from it: if markup would let a reader believe a person formed an opinion and none did, the markup is wrong, however accurate the number inside.

Our own answer measurements query an API too, so the same rules bind us: every row names the model, the mode and the date, and describes what that model said that day, not what a person saw. Seven matched questions on two dates is published in that form. The free-tool version of this problem is covered in what free AI visibility checkers really measure.

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Do AI visibility tools measure what people see in ChatGPT?

No. They query provider APIs, and an API is not the consumer app. ChatGPT adds retrieval, tools, personalisation and memory that the raw API does not reproduce, and AI Overviews is a different system again.

Are AI visibility scores fake?

The recorded answers are real model output. What is unsupported is the label placed on them, because the surface, the prompt sample and the moment all differ from what the number claims to describe.

What makes an AI visibility number checkable?

Naming the model, the mode and the date, reporting presence or absence rather than a rank, and publishing the churn between dated runs so a reader can see how stable the result is and re-run it.

Is share of voice a real metric for AI search?

It is real about the prompt list it was computed over, on the day it was computed. It is not a property of a brand, because nobody holds the distribution of what buyers actually ask and the same prompt returns different answers between runs.

Is an API-based AI visibility score ever useful?

Yes, as a change detector. Run against a fixed prompt list with the model and mode pinned, on a schedule, a series of readings can show that something moved. The absolute percentage on its own means little.

What part of AI visibility can be measured directly?

Whether your site is available as an input: whether a response arrives for an AI crawler, whether the bytes carry readable text, and whether there is a passage worth quoting. That shows availability, not citation.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works