CrawlCheck

Agent view · Findings · 2026-08-20

AI visibility scores measure an API, not ChatGPT

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/ai-visibility-scores-measure-an-api-not-chatgpt: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/ai-visibility-scores-measure-an-api-not-chatgpt>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes14,786
words2,148
sha-256ae74010bd31a2015808e032b7c8e875a695a38dd96b5bbabe42f29674161ec11
repr-digestsha-256=:rnQBC9MaIBWAjgMrfI6HWmlaON2Wtbur5C8pZ0Fh7BE=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311494;expires=1795315094;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="cdrt05Z-fuW1OG58lC5RRA";tag="crawlcheck-page"
signaturesig1=:V89PMQndmliPykhM8Jthrx3usaYJBTohHkGOgq2AviQhXFcj9H88TtP4f4B11Kh8eunG6QzA3Bsc2jTlyfPrAQ==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Findings · 2026-08-20 · By [VSNARY | Emmanuel Orta](/author)

# AI visibility scores measure an API, not ChatGPT

The answers these tools record are real. The label on top — “your visibility in AI search” — describes something that was never measured.

They measure something real: what a model API returned for a chosen prompt list on a given day. What they usually claim is different — your visibility in ChatGPT or AI Overviews. Three substitutions sit between the two. An API is not the consumer app, which adds retrieval, tools, personalisation and memory. A hand-picked prompt list is not a property of your brand. And one answer is one draw from a distribution that changes between runs. The number is not fake; the sentence wrapped around it is.

On this page

- What sits between the API answer and the claim?
- How do you tell a measurement from a label?
- Is the API number useless, then?
- What can be measured directly?
- Where we caught the same error in our own markup

An AI visibility tool asks the Gemini API *who is the best tree service in Denver*, records the answer, counts how often your brand appears and reports a percentage. Every step is real; the model genuinely produced those words. What is not real is the headline wrapped around the number: **“your visibility in AI search.”**

## What sits between the API answer and the claim? #

The number is real. The sentence wrapped around it crosses three gaps.

| What was measured | What the label says | Why they differ |
| --- | --- | --- |
| The Gemini or OpenAI API | your visibility in ChatGPT or AI Overviews | ChatGPT adds [retrieval](/glossary/retrieval), tools, personalisation and memory the raw API does not reproduce; AI Overviews is a different system again, inside a results page |
| A hand-picked prompt list | a property of your brand | nobody holds the distribution of what buyers actually ask; change the list and the number changes |
| One answer, on one day | a stable [share of voice](/glossary/share-of-voice) | the same prompt gives different answers between runs, and larger differences after a model update |

None of this is fraud. There is no public API for the consumer surfaces, so every vendor measures what it can reach and labels it as what you asked about. The gap is in the rename, and the rename is easy to spot: it always uses a word the method never did.

## How do you tell a measurement from a label? #

Ask what would have to be true for the number to be wrong, and how you would find out. If there is no answer, it is not a measurement. What survives that test is narrow: name the model, the mode (grounded or from weights) and the date, report presence or absence rather than a rank, and publish the churn between dated runs.

| Ask the vendor | A straight answer sounds like |
| --- | --- |
| Which surface? | “the Gemini API”, not “AI search” |
| Which model, and which mode? | a version string, and grounded or from weights |
| Which date? | a date, not “recently” |
| What was the sample? | the prompt list, or the [URL](/glossary/url) and the client identity |
| Can I re-run it? | the method, in enough detail to disagree with |

The first four are facts a vendor either has or does not. The fifth decides whether you bought a measurement or a label.

## Is the API number useless, then? #

No. Run against a fixed prompt list, with the model and mode pinned, on a schedule, it is a reasonable change detector. The absolute percentage means little; a series can tell you something moved, and you can then ask whether the prompt list, the model version or the mode changed. That is a useful signal and a much smaller claim than the one usually printed above it. Measuring a claim properly would need the consumer surface itself, the documents it retrieved for that user at that moment, and attribution for each sentence of the answer. Nobody outside the providers holds any of the three.

## What can be measured directly? #

The retrieval side: not whether an assistant named you, but whether it could have. That is a fetch, and a fetch can be repeated. Three yes-or-no questions with a number behind each: **reach** (does a response arrive for a non-browser client), **read** (is there readable text in the bytes before anything runs) and **quote** (is there a self-contained passage worth lifting). CrawlCheck answers them by fetching the page as each named AI crawler and comparing the replies. The result is a status code, a ratio and a diff — evidence the input was available, never evidence of [citation](/glossary/citation). [How to measure AI visibility yourself](/guides/how-to-measure-ai-visibility) walks through the same checks by hand.

## Where we caught the same error in our own markup #

Our directory publishes a measured score for each listed vendor site. For a while its [structured data](/glossary/structured-data) emitted each score as a schema.org `Review` with a `reviewRating`. That was this exact substitution: a review says a person formed an opinion, and nobody did — a scanner computed a number. It now emits an `Observation`, with the score as measured values and a link to the evidence report it came from. The rule we took from it: **if markup would let a reader believe a person formed an opinion and none did, the markup is wrong**, however accurate the number inside.

Our own answer measurements query an API too, so the same rules bind us: every row names the model, the mode and the date, and describes what that model said that day, not what a person saw. [Seven matched questions on two dates](/blog/seven-matched-questions-ai-citations-october-2026) is published in that form. The free-tool version of this problem is covered in [what free AI visibility checkers really measure](/guides/free-ai-visibility-checkers-what-they-measure).

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fai-visibility-scores-measure-an-api-not-chatgpt%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fai-visibility-scores-measure-an-api-not-chatgpt%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=AI%20visibility%20scores%20measure%20an%20API%2C%20not%20ChatGPT)[Email](mailto:?subject=AI%20visibility%20scores%20measure%20an%20API%2C%20not%20ChatGPT&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fai-visibility-scores-measure-an-api-not-chatgpt)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Do AI visibility tools measure what people see in ChatGPT?

No. They query provider APIs, and an API is not the consumer app. ChatGPT adds retrieval, tools, personalisation and memory that the raw API does not reproduce, and AI Overviews is a different system again.

### Are AI visibility scores fake?

The recorded answers are real model output. What is unsupported is the label placed on them, because the surface, the prompt sample and the moment all differ from what the number claims to describe.

### What makes an AI visibility number checkable?

Naming the model, the mode and the date, reporting presence or absence rather than a rank, and publishing the churn between dated runs so a reader can see how stable the result is and re-run it.

### Is share of voice a real metric for AI search?

It is real about the prompt list it was computed over, on the day it was computed. It is not a property of a brand, because nobody holds the distribution of what buyers actually ask and the same prompt returns different answers between runs.

### Is an API-based AI visibility score ever useful?

Yes, as a change detector. Run against a fixed prompt list with the model and mode pinned, on a schedule, a series of readings can show that something moved. The absolute percentage on its own means little.

### What part of AI visibility can be measured directly?

Whether your site is available as an input: whether a response arrives for an AI crawler, whether the bytes carry readable text, and whether there is a passage worth quoting. That shows availability, not citation.

## Related findings

[2026-08-28 How to measure AI visibility yourself: three checks Measure AI visibility in three steps: check crawlers can read your site, verify which crawlers really arrive by IP, then log answe](/guides/how-to-measure-ai-visibility)[2026-08-29 Missing referrer is not direct traffic: 93% of our arrivals sent none We measured arrivals on five sites we operate with our own first-party beacon. 92.9% arrived with no referrer. That number decides](/blog/ninety-three-percent-of-our-arrivals-sent-no-referrer)[2026-08-28 Does llms.txt work? What is known and how to test it Does llms.txt work? What is verifiable, what we measured on our own site, and the server-log test with IP verification that answer](/guides/does-llms-txt-actually-work)

Where this is used

Guides that cite this finding:

- [AI visibility tools compared: what each one measures](/guides/ai-visibility-tools-what-each-one-measures)
- [GEO vs AEO vs LLM SEO: the terms, and what is measurable](/guides/geo-aeo-llm-seo-what-the-terms-mean)
- [How to measure AI visibility yourself: three checks](/guides/how-to-measure-ai-visibility)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [SDK](/sdk) · [Certificates](/certificates) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [MCP](/registry/mcp) · [MCP: Report](/registry/mcp/report) · [cnrs.fr](/registry/cnrs.fr) | **Glossary:** [AI classifiers: Scoring](/glossary/ai-classifiers/scoring) · [Document ingestion](/glossary/document-ingestion) · [Connection timeout](/glossary/connection-timeout) · [Redirected internal link](/glossary/redirected-internal-link) · [preflight](/glossary/preflight) · [Markdown negotiation](/glossary/markdown-negotiation)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /blog/ai-visibility-scores-measure-an-api-not-chatgpt.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.