Exclusive finding · Findings · 2026-09-12 · By VSNARY | Emmanuel Orta

Google-Extended has no user agent. 1,582 requests used it

Google's documentation says Google-Extended, a robots.txt control token, has no HTTP user-agent string. Over seven days, 1,582 requests reached sixteen of our sites carrying it anyway.

Between 5 and 11 September 2026 we read Cloudflare's own bot-verification verdict on every request that reached sixteen sites we operate on Cloudflare. Not a guess from a user-agent string, and not an outside-in probe: the verdict the network that served the request already recorded. Two counts per crawler name, over the same requests, in the same window - how many Cloudflare verified, and how many carried the same name and were not verified.

Eleven names came back with a rate worth publishing. Five came back having never been verified once. One of those five should not exist as a user-agent at all.

The name that is not a crawler #

Google-Extended is the robots.txt token that governs whether content Google has already crawled may be used to train and ground Gemini. It is a policy control, not a fetcher. Google's crawler documentation is explicit about it: Google-Extended does not have a separate HTTP request user agent string, crawling is done with existing Google user agent strings, and the robots.txt token is used in a control capacity. How to use the token is covered in Google-Extended and Applebot-Extended: opt out of AI training and stay in search.

So there is nothing to verify, because nothing is supposed to arrive. Over the window, 1,582 requests reached these sites carrying Google-Extended in the user-agent. Whatever sent them, it was not Google using a documented identity, because Google documents that this identity is never sent.

That is a different statement from the one our page was making. A rate of 0% verified reads as an accusation about an operator's fleet. The true statement here is narrower and stronger: a name that its owner says never appears on the wire appeared 1,582 times.

What the other four mean, which is not the same thing #

PerplexityBot arrived 4,893 times and was verified zero times. That is not evidence that 4,893 requests were forged. Cloudflare removed Perplexity from its Verified Bots programme in August 2025, naming PerplexityBot and Perplexity-User as the identifiers. Once a crawler is outside the programme there is no verification path at all, so every request under that name lands unverified by construction - a fact about the reference data, not about the requests.

CCBot and Claude-SearchBot are in the same position from where we stand: never verified once, across sixteen sites and seven days. We have no external evidence about either, and we are not going to invent any. What we can say is that we have never seen a verification succeed for those names here, which is exactly the sentence the page now prints instead of a percentage.

The rates that are real #

For the eleven names where verification demonstrably works, the spread is wide and it does not line up with how people talk about these crawlers.

Even Googlebot, the most verifiable crawler on the internet, sits at 51.5%. Roughly half the requests arriving at these sites under Googlebot's name were not verified as Googlebot by Cloudflare. That is the number most site owners would be most surprised by, and it is the one we very nearly failed to publish at all.

Our own figures were publishing a false zero #

Before 12 September 2026, our public crawler figures left out every verified request from crawlers that Cloudflare classes as search engines or SEO tools rather than AI.

The result: 19 of 30 rows read 0% verified, and Googlebot was one of them. Cloudflare files Googlebot as a search engine crawler, not as AI, so not one of its 5,284 verified requests was counted. The same went for bingbot, AhrefsBot, SemrushBot, MJ12bot, Baiduspider and YandexBot - every one of them verified thousands of times, every one of them published at zero.

Nobody was reading those figures yet, which is the only reason this is a note rather than an apology. It is still the worst kind of defect we ship: a number that looks measured, on a page whose entire proposition is that the numbers are measured. We have done this to ourselves before, in the other direction, and counted our own traffic as forged.

A rate needs a verification path to exist #

Counting both sides the same way was the easy half. The harder half is that 0% says two opposite things and cannot tell you which. Either the crawler is verifiable and not one request passed, or there is no verification path for that name and nothing could ever pass. Printed as a percentage, a reader takes the first reading every time.

So a name we have never seen verified, across sixteen sites and seven days, no longer gets a percentage. Those five rows now carry the counts and the reason, alongside the operator’s own published statement where one exists. We do not know that Cloudflare cannot verify CCBot; we know we have not seen it happen. That distinction is the same one every unscored row on a report rests on.

What this does not claim #

These are sixteen of our own sites, measured because we operate them, not a sample of the web. Unverified is not forged: a legitimate crawler behind a proxy, or one whose ranges Cloudflare has not adopted, lands in the same bucket as an impostor. The unverified counts come from Cloudflare's daily lists of the most common user-agent strings on each site, so a rare impostor on a busy site can fall outside the list.

What survives all of that is narrow and worth saying anyway. Half the traffic arriving as Googlebot was not verified as Googlebot. And 1,582 requests introduced themselves with a name that the company that owns it says it never sends. If you want to see the current figures rather than these, they are on the crawler reference, recomputed from the same records. To check a single request yourself, see how to verify Googlebot, Bingbot and AI crawlers by IP address.

Questions this post answers

Does Google-Extended have its own user agent?

No. Google documents Google-Extended as a robots.txt control token only. Crawling is done with Google's existing crawler user agents, so a request whose user agent says Google-Extended is not Google using a documented identity.

Should I block requests that say Google-Extended?

Blocking them costs Google nothing, because Google never sends that name. To stop Google using your content for Gemini, add a Google-Extended group to robots.txt; that is the control the token exists for.

Does 0% verified mean every request was forged?

Not necessarily. It can mean no verification path exists for that name, as with PerplexityBot after Cloudflare removed Perplexity from its Verified Bots programme. A 0% figure only means something when the crawler can be verified at all.

Is unverified traffic the same as forged traffic?

No. A legitimate crawler behind a proxy, or one whose ranges Cloudflare has not adopted, is unverified without being forged. Unverified means the claim could not be confirmed, not that it was proven false.

What did CrawlCheck get wrong in its own figures?

For a period, its public crawler figures left out verified requests from search engine and SEO crawlers, so Googlebot, bingbot and others showed 0% verified. The counting was corrected and the page now shows the measured rates.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works