Findings · 2026-11-06 · By VSNARY | Emmanuel Orta
Our AI accuracy check ran for six days and measured nothing
The daily job that asks Gemini what it tells customers about each business got a 403 on every call from 31 October to 6 November 2026. The job reported success. What broke, why nothing alerted, and what a wrong answer looked like when it came back.
From 31 October to 6 November 2026 every call CrawlCheck made to Google's Gemini API returned 403 PERMISSION_DENIED with the reason API_KEY_SERVICE_BLOCKED. Google now requires a Gemini key to be an auth key bound to a service account; the standard API key in use could no longer add the Gemini service. The daily check recorded the failure as 'no reading' and finished without error, so nothing alerted. The fix was a new bound key set as a Worker secret, a billing move, and a change so that a 403 falls through to the next key and carries Google's reason.
Once a day CrawlCheck asks an AI model what it would tell a customer about each business it watches: the phone number, the rating, the review count, the hours. Each answer is checked against the business's own page and marked right or wrong. It is the closest thing we have to a measurement of what an AI assistant is actually saying about a small business.
On 6 November the panel showed one wrong answer for a fencing business, dated 30 October: the model said four stars from three reviews, the site shows a 5.0 rating from 136 Google reviews. That row was a week old. Nothing newer had been written. The check had not produced a reading since 31 October.
What the 403 said #
Every Gemini call since 31 October had returned 403 PERMISSION_DENIED with the reason API_KEY_SERVICE_BLOCKED. Google had changed the rule: a key used for Gemini now has to be an auth key bound to a service account, and the ordinary API key the project had used since September could no longer be granted the Gemini service. The key still worked for other Google APIs. From Gemini's side it was blocked, and it said so in every response.
| Date | What the job did | What it recorded |
|---|---|---|
| To 30 October | Called Gemini, checked each answer | 11 businesses, readings written daily |
| 31 October to 6 November | Called Gemini, received 403 on every call | 'No reading' for every business; run marked complete |
| 6 November, 2:40 PM MST | New bound key live, billing moved, 11 businesses checked | 11 checked, 0 wrong |
Why nothing alerted #
The check treats an unanswered question as a gap, not a failure. That is right for one business on one day, since a model can decline a question. It is wrong for every business on six consecutive days. No rule looked at the shape of the gaps, and the run's own status was 'complete' because it had done everything it was asked to do, which was to try.
We keep writing this sentence about ourselves: the process ran, every check confirmed the process ran, and nobody opened the output. The crawler counter fixed the same morning failed the same way.
Three things had to change to restore it #
First, a service account with no roles and a new auth key bound to it, restricted to the Gemini API. Second, the key had to reach two Workers as a secret. Cloudflare's dashboard Deploy button did not send the request, twice, so the key now travels from a GitHub repository secret to the Workers through a workflow that never prints it. Third, the first successful authentication produced a 402: the project was billed to an account the owner could not administer. It now bills to one he does.
In the code, a 403 falls through to the next bound Google key instead of ending the attempt, and the recorded error names Google's reason and the project. A six-day run of identical 403s would now read as what it is.
The wrong answer that was waiting #
The 30 October row was real and worth having. The model's 'four stars from three reviews' was the business's Trustpilot profile, where three five-star reviews, one of them by the owner, produce a TrustScore of 4.0. The site's own page says 5.0 from 136 Google reviews. The model picked the smaller, older source.
The fix for that is on the business side, not ours: the machine-readable file each site serves now states its Google rating and review count with the date it was read, so a model that reads the site has the current figure in plain text. On the next run the model quoted the site's own reviews note. We had measured four models against these businesses in October; this is the follow-through on one of them.
The check has run clean since. The six missing days stay missing.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
Why did Gemini return 403 API_KEY_SERVICE_BLOCKED?
Google now requires a Gemini API key to be an auth key bound to a service account. A standard API key can no longer be granted the Gemini service, so every call with the old key was refused with that reason.
How long was CrawlCheck's AI accuracy check down?
Six days, 31 October to 6 November 2026. Every call failed; the job recorded 'no reading' for each business and finished without an error.
Why did no alert fire?
The check treated an unanswered question as a gap rather than a failure, which is correct for one business on one day. Nothing looked for every business failing on consecutive days, and the run's own status was complete.
What did the one wrong answer get wrong?
A fencing business's rating. The model quoted a Trustpilot profile, 4.0 from three reviews, while the site shows 5.0 from 136 Google reviews. The site's machine-readable file now states the Google rating and review count with the read date.
Is the check running now?
Yes. On 6 November 2026 at 2:40 PM Mountain Time a bound key went live, 11 businesses were checked and 0 answers were wrong. A 403 now falls through to the next key and records Google's reason.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit, and placement is never for sale.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.