Agent view · Findings · 2026-11-06

Our AI accuracy check ran for six days and measured nothing

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/our-ai-accuracy-check-ran-for-six-days-and-measured-nothing: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

Findings · 2026-11-06 · By [VSNARY | Emmanuel Orta](/author)

# Our AI accuracy check ran for six days and measured nothing

The daily job that asks Gemini what it tells customers about each business got a 403 on every call from 31 October to 6 November 2026. The job reported success. What broke, why nothing alerted, and what a wrong answer looked like when it came back.

From 31 October to 6 November 2026 every call CrawlCheck made to Google's Gemini API returned 403 PERMISSION_DENIED with the reason API_KEY_SERVICE_BLOCKED. Google now requires a Gemini key to be an auth key bound to a service account; the standard API key in use could no longer add the Gemini service. The daily check recorded the failure as 'no reading' and finished without error, so nothing alerted. The fix was a new bound key set as a Worker secret, a billing move, and a change so that a 403 falls through to the next key and carries Google's reason.

On this page

- What the 403 said
- Why nothing alerted
- Three things had to change to restore it
- The wrong answer that was waiting

Once a day CrawlCheck asks an AI model what it would tell a customer about each business it watches: the phone number, the rating, the review count, the hours. Each answer is checked against the business's own page and marked right or wrong. It is the closest thing we have to a measurement of what an AI assistant is actually saying about a small business.

On 6 November the panel showed one wrong answer for a fencing business, dated 30 October: the model said four stars from three reviews, the site shows a 5.0 rating from 136 Google reviews. That row was a week old. Nothing newer had been written. The check had not produced a reading since 31 October.

## What the 403 said #

Every Gemini call since 31 October had returned `403 PERMISSION_DENIED` with the reason `API_KEY_SERVICE_BLOCKED`. Google had changed the rule: a key used for Gemini now has to be an auth key bound to a service account, and the ordinary [API key](/glossary/api-key) the project had used since September could no longer be granted the Gemini service. The key still worked for other Google APIs. From Gemini's side it was blocked, and it said so in every response.

| Date | What the job did | What it recorded |
| --- | --- | --- |
| To 30 October | Called Gemini, checked each answer | 11 businesses, readings written daily |
| 31 October to 6 November | Called Gemini, received 403 on every call | 'No reading' for every business; run marked complete |
| 6 November, 2:40 PM MST | New bound key live, billing moved, 11 businesses checked | 11 checked, 0 wrong |

## Why nothing alerted #

The check treats an unanswered question as a gap, not a failure. That is right for one business on one day, since a model can decline a question. It is wrong for every business on six consecutive days. No rule looked at the shape of the gaps, and the run's own status was 'complete' because it had done everything it was asked to do, which was to try.

We keep writing this sentence about ourselves: the process ran, every check confirmed the process ran, and nobody opened the output. The [crawler counter](/blog/our-crawler-counts-missed-45-percent-of-hits) fixed the same morning failed the same way.

## Three things had to change to restore it #

First, a service account with no roles and a new auth key bound to it, restricted to the Gemini API. Second, the key had to reach two Workers as a secret. Cloudflare's dashboard Deploy button did not send the request, twice, so the key now travels from a GitHub repository secret to the Workers through a workflow that never prints it. Third, the first successful authentication produced a 402: the project was billed to an account the owner could not administer. It now bills to one he does.

In the code, a 403 falls through to the next bound Google key instead of ending the attempt, and the recorded error names Google's reason and the project. A six-day run of identical 403s would now read as what it is.

## The wrong answer that was waiting #

The 30 October row was real and worth having. The model's 'four stars from three reviews' was the business's Trustpilot profile, where three five-star reviews, one of them by the owner, produce a TrustScore of 4.0. The site's own page says 5.0 from 136 Google reviews. The model picked the smaller, older source.

The fix for that is on the business side, not ours: the machine-readable file each site serves now states its Google rating and review count with the date it was read, so a model that reads the site has the current figure in plain text. On the next run the model quoted the site's own reviews note. We had [measured four models against these businesses in October](/blog/what-four-ai-models-told-customers-about-our-three-businesses); this is the follow-through on one of them.

The check has run clean since. The six missing days stay missing.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-ai-accuracy-check-ran-for-six-days-and-measured-nothing%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-ai-accuracy-check-ran-for-six-days-and-measured-nothing%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Our%20AI%20accuracy%20check%20ran%20for%20six%20days%20and%20measured%20nothing)[Email](mailto:?subject=Our%20AI%20accuracy%20check%20ran%20for%20six%20days%20and%20measured%20nothing&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-ai-accuracy-check-ran-for-six-days-and-measured-nothing)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Why did Gemini return 403 API_KEY_SERVICE_BLOCKED?

Google now requires a Gemini API key to be an auth key bound to a service account. A standard API key can no longer be granted the Gemini service, so every call with the old key was refused with that reason.

### How long was CrawlCheck's AI accuracy check down?

Six days, 31 October to 6 November 2026. Every call failed; the job recorded 'no reading' for each business and finished without an error.

### Why did no alert fire?

The check treated an unanswered question as a gap rather than a failure, which is correct for one business on one day. Nothing looked for every business failing on consecutive days, and the run's own status was complete.

### What did the one wrong answer get wrong?

A fencing business's rating. The model quoted a Trustpilot profile, 4.0 from three reviews, while the site shows 5.0 from 136 Google reviews. The site's machine-readable file now states the Google rating and review count with the read date.

### Is the check running now?

Yes. On 6 November 2026 at 2:40 PM Mountain Time a bound key went live, 11 businesses were checked and 0 answers were wrong. A 403 now falls through to the next key and records Google's reason.

## Related findings

[2026-10-14 Our directory froze for 20 hours and logged no error Our directory index stopped rebuilding for 20 hours on 13 and 14 October 2026 and eight cards opened 404s. What visitors saw, the](/blog/a-cron-job-ran-one-minute-late-and-our-directory-froze-for-20-hours)[2026-08-20 Why caching only successes kept our scans slow Scans of small sites ran slow because a 'no data' answer was never remembered. Caching the miss cut a repeated 8.2 s lookup to 3 m](/blog/cache-your-misses-a-lookup-that-only-caches-on-success)[2026-08-26 Lighthouse scores vary between runs: two runs, 225% apart Two Lighthouse runs of an unchanged page, minutes apart: blocking time 2,643 ms then 812 ms. How to find your noise floor before a](/blog/one-lighthouse-run-cannot-support-a-finding-225-percent-variance)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit, and placement is never for sale.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [MCP](/registry/mcp) · [calibreapp.com](/registry/calibreapp.com) · [docusign.com](/registry/docusign.com) · [grain.com](/registry/grain.com) · [lg.com](/registry/lg.com) | **Pages:** [Listing check](/listing-check) · [Entity map validator](/entity-map-validator) · [Status](/status) | **Glossary:** [Status message](/glossary/status-message) · [content-change clock](/glossary/content-change-clock) · [Content-Language header](/glossary/content-language) | **Directory:** [Category: AI video image](/directory/category/ai-video-image)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Report an error](/report-error)

Same body as a file: /blog/our-ai-accuracy-check-ran-for-six-days-and-measured-nothing.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.