Findings · 2026-08-13 · By VSNARY | Emmanuel Orta · 0 views

Which AI crawlers actually visit a small business site

Not a probe: real, unasked-for visits to sites we operate, each checked against the operator's published IP ranges. Which AI crawlers turned up, how often, and which were forged.

Almost everything written about AI crawlers is a probe: someone sends a request wearing a crawler’s name and reports what came back. Useful, but it is not the same question as who actually turns up.

We started recording that on a handful of sites we operate — counts only, no address stored, no cookie set. The first day produced two things worth saying, and one sentence we got wrong.

Bytespider arrived most, and went for a PDF

ByteDance’s crawler made more visits than any other named agent, and the path it requested most was not the homepage. It was a PDF sitting in an uploads folder.

Most operators we talk to have never considered Bytespider at all. It is not in the usual list, and it is not what people mean when they say AI crawler.

A correction to this post

As first published, this paragraph said those visits were every one of them verified against the ranges its operator publishes. That was wrong, and our own site says so in four other places: ByteDance publishes no address ranges. There is nothing to check a Bytespider request against. Every one of those visits is recorded here as unverifiable — which is not an accusation and not a clearance, and is precisely the state the rest of this post exists to explain.

Leaving the sentence up would have been worse than the error. A tool whose argument is a user-agent is a claim, not an identity cannot afford a post that treats an unverifiable name as verified because the crawler was interesting.

The same name arrives from two different places

One agent name showed up both verified and unverified on the same day. The verified visits came from the operator’s published ranges. The unverified ones came from a cloud network — in that instance, our own testing, which appears in the data as unverified because it is.

That is the point of separating the two columns. Unverified is not proof of an impostor: plenty of legitimate traffic arrives from ranges nobody publishes. But the name in a user-agent string is a claim, and it is checkable.

Which operators can be checked at all

This is the fact that decides how every row above should be read, and it splits the named crawlers into two groups that no access log distinguishes.

A name in the right column can never be marked verified here, however well-behaved the crawler is. That is a limit of the evidence, not a judgement about the operator.

The consequence is uncomfortable for anyone quoting a forgery rate: the honest denominator is verifiable claims, not all claims. Divide by everything and you have counted no feed published as fine, which flatters the number by exactly the traffic you know least about.

The visits above were counted on the day this was written. The block below is the same adjudication running now, on requests this site received while you were reading it — every claim checked against the range its operator publishes.

Forged crawler identities

23% of the requests that named themselves as a known crawler here were not that crawler. 1369 of 5940 checkable claims came from an address outside the range the operator publishes.

A user-agent is a claim, not an identity. Verified means the source IP sits inside a range the operator publishes — OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish one. Unverifiable is not forgery: some operators publish no range at all, so their requests can be neither confirmed nor accused, and they are excluded from the rate rather than counted against it. This is traffic to this site only, and this site is small — it is a floor on the problem, not a survey of the web.

What we changed because of this

The correction above was not just an edit to a paragraph. The rule it broke is now enforced in the code rather than remembered: a crawler with no readable feed can only ever be recorded as unverifiable, and no code path can promote it. The full list of which operators publish ranges lives on the crawler reference and is read from the same map the verifier uses, so the page and the check cannot drift apart.

The published rate follows the same rule. Unverifiable hits are excluded from the denominator and stated as excluded, which is the difference between a forgery rate and a number that sounds like one. Paste your own log lines into the free verifier and it will answer the same three ways: verified, forged, or unverifiable because nobody publishes a range to check against.

Why your access log will not tell you this

A log records the user-agent string. It does not record whether that string was earned. Verification means matching the request against the address ranges the operator publishes — a column no access log carries, and the difference between GPTBot visited you and something called itself GPTBot.

There is a second column the default log format throws away: the Host header. Keep it and you can see whether a crawler prefers your apex or your www, which is a question with no published answer anywhere and a real effect on which copy of a page gets indexed. We record it on every hit for exactly that reason — the same reasoning as measuring what each crawler is served rather than assuming they all get the page you see.

One more thing the two-column split makes visible, and it is the reason the unverifiable column is not a rounding error: several of the highest-volume named agents arriving here belong to operators with no feed at all. Whatever share of your traffic those names represent is traffic whose identity is structurally uncheckable by anyone, using any tool, including this one. A vendor who reports it as verified is not measuring harder than us; they are guessing, and calling the guess a number.

The counts here are a sample from a few sites, not a census of the web, and they are published as such.

How to read your own log the same way

Pull the user-agent strings from a week of access logs, count them, then verify the ones that claim to be a crawler against the operator's published ranges. The claim column and the verified column will disagree, and the gap is the finding:

grep -i 'gptbot\|claudebot\|bytespider\|perplexitybot' access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

Take the addresses that appear most and check each against the JSON feed the operator publishes; the crawler list links every feed. An address outside every range is a claim you cannot verify, which is different from a forgery, and the report should keep those apart.

What has changed since this was written

Verified-only counting is now the default across every telemetry number this site publishes, and the beacon that records arrivals on the sites we operate stores a referrer hostname and nothing else. The most recent figures are in how much AI crawler traffic is forged; the definitions are verified crawler and retrieval crawler in the glossary.

Questions this post answers

Which AI crawlers actually visit a small business site?

Measured from real unsolicited visits, Bytespider arrived most often and went for a PDF rather than the pages. ByteDance publishes no address ranges, so those visits are recorded as unverifiable rather than verified.

Can the same crawler name arrive from two places?

Yes, and it did. One name arrived from addresses inside the operator's published range and from addresses outside it in the same period.

Why will my raw access log not tell me this?

Because the log records the claimed name. Without checking each address against the operator's feed, the genuine and the forged look identical.

Related findings

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works