CrawlCheck

Findings · 2026-09-23 · By · 0 views

The one US directory Google lists as a data provider refused our scanner. A browser got the answer in one second.

Data Axle returns a 10-byte 403 to every datacenter address we have. The reading that matters for a local business was structurally out of reach of a server-side scanner, so the scanner now accepts it from the owner's own browser and parses it itself.

Data Axle's listing search refuses every datacenter address with a 403, so a server-side scanner cannot read it. CrawlCheck now accepts the search page from the site owner's own browser, stores the raw page, and parses it with the same reader it uses for every directory. Across nine sites in one portfolio, one was listed and eight were absent.

Share of all scans carrying each finding named aboveNAP_UNDECLARED_LISTING5.7%Share of all scans carrying eachfinding named aboveNAP_UNDECLARED_LISTING5.7%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

When we extracted the provider list Google's Maps stack ships with, the 793-name enum had no Yelp, no Foursquare and no Yext in it. It had one US business-data aggregator: Data Axle. So the row that matters most for a local business's presence in Google's data supply is the Data Axle row. That row had been reading refused (403) — our limit, not an absence on every scan of every site since the day we added it.

What the refusal looked like #

Data Axle's public listing search answers a phone-number query with a page that either shows the listing or says it couldn't find a match. From the scanner it never showed either. It answered HTTP 403 with a ten-byte body, Forbidden, in about 1.2 seconds when it was quick and 8,214 ms when it was not. The slow refusal was the whole listing check's budget on one site, so the row was given four seconds of its own and stayed report-only: a directory that never answers cannot be allowed to move a grade.

The scanner's first assumption was the user-agent. It was wrong. From a single clean address, a Chrome user-agent and a Google Apps Script user-agent both got the page; from a datacenter address the same Chrome string got the 403. From an address that rotated between clean and blocked, the same request alternated 200 and 403. The wall is address reputation, and every address a Cloudflare Worker or a cloud container can send from is on the wrong side of it.

The relay that did not help #

The listing check already had an egress relay: a script hosted on Google's own infrastructure that fetches on the scanner's behalf, built because YellowPages and DexKnows sometimes throttle the Worker's addresses. We allowlisted Data Axle on it. Six of six tries came back with the same ten-byte 403. Google's egress addresses are datacenter addresses too. A relay moves the request to a different cloud; it does not make it residential.

VantageAddress classData Axle answered
Cloudflare Worker (the scanner)datacenter403, 10 bytes, every time
Google Apps Script relaydatacenter403, 6 of 6
Analysis containerdatacenter, rotating200 or 403 depending on the address drawn
The site owner's Chromeresidential200, 9 of 9, about a second each

The vantage that answers #

The scanner already had one route for this shape of problem. For profile pages that refuse crawlers, a licensed site owner can file the page as their browser saw it, and the entity corroboration row is marked owner-browser and consulted only where the scanner's own fetch was unverifiable. The Data Axle row now works the same way. The owner's browser fetches the search page for the phone number the site declares, posts the raw page to the scanner, and the scanner parses it with the same reader it has always used for that directory. The browser is never asked whether the business is listed. It carries the page; the reading happens on our side.

That boundary is the same one the observation network draws for crawler logs: the sender is never trusted for the field that matters. A client that could assert listed: true would turn a measurement into a claim. A client that can only hand over a page cannot.

An owner-browser page is kept for seven days and takes precedence over the scanner's own fetch on that URL. The row states its vantage and the date it was filed, so a reader can see that the answer came from a browser on a given day rather than from the scan that rendered the report.

Nine sites, one listing #

The first run was the portfolio the scanner was built against: nine service and retail businesses in the Denver area, each with a phone number declared in structured data. Data Axle listed one. The other eight returned the directory's own no-match page, which the scanner records as the directory's answer, not as a failure to reach it. Two outcomes that look identical to a reader ("nothing shown") are recorded as two different things: absent, and unreached.

OutcomeSitesRecorded as
Listing found, name and address parsed1found, via owner-browser
Data Axle's no-match page8no listing carries this phone (the directory's no-match page)
Not reached0refused — our limit, not an absence

The one listing was correct: the name Data Axle holds matches the site, and the street address matches the one the site publishes. That is the case where a directory row adds nothing but confidence. The eight absences are the useful result. Listing with Data Axle is free, it is the aggregator Google's own enum names, and none of the eight businesses had done it. A directory check that could not read the directory would have shown all nine as our limit and told the owner nothing.

What this does not prove #

A Data Axle listing is presence in one supplier's database. It is not a ranking factor, it is not a citation in the review-site sense, and the scanner does not score it. The row is reported, not scored, and it stays that way until it has been calibrated on enough sites to know what a typical business looks like. The undeclared-listing rate the scanner does score, 5.7% of scans, comes from the directories it can read directly.

It also does not prove the eight businesses are invisible to Google. Google's data comes from many providers, and a site's own structured data is one of them. What it proves is narrower and more useful: the one US aggregator in Google's provider list had no record of eight of nine businesses, and now the owners know.

Findings behind this #

The owner-browser route is available to any licensed site through the CrawlCheck extension, which reads the phone number the current page declares, fetches Data Axle from the visitor's own browser and files the page. The provider enum this started from is published in full in the post that extracted it; the corpus rates every finding code carries are on the dataset page.

Every figure above came out of this scanner.

Point it at your own domain and see the same measurements, free.

Scan a domain — free

The main product

Found this on your own site? We fix it for $749.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Questions this post answers

Why can't a scanner read Data Axle's listing search?

Data Axle answers every datacenter address with a 403 and a ten-byte body. It is address reputation, not the user-agent: the same request from a residential address gets the page. A relay hosted on Google's infrastructure got the same refusal, because its egress addresses are datacenter addresses too.

How does the owner-browser vantage work?

The site owner's browser fetches the Data Axle search page for the phone number the site declares and posts the raw page to CrawlCheck, which parses it with the same reader it uses for every directory. The browser never says whether the business is listed; it only carries the page.

Does a Data Axle listing affect the CrawlCheck grade?

No. The row is reported, not scored, and stays that way until it is calibrated on enough sites. The report shows the vantage and the date the page was filed.

What did the first run find?

Across nine businesses in one portfolio, Data Axle listed one and returned its no-match page for eight. The eight are recorded as absent, which is the directory's answer, not as unreached.

Why does Data Axle matter more than Yelp here?

The provider list Google's Maps stack ships with names one US business-data aggregator, Data Axle, and does not name Yelp, Foursquare or Yext. Presence in the supplier Google names is a narrower, more checkable fact than presence on a review site.

Related findings

How anything measured in this article was measured15client identitiesone second, one address5machine filesapex and www114named agentsresolved from robots.txt24sections scoredreach, read, quoteHow anything measured here was measured15 client identities5 machine files114 named agents24 sections scoredone second, one addressapex and wwwresolved from robots.txtreach, read, quote
No account, nothing installed, and the same sequence on every domain — which is what makes one scan comparable to another. Run it on your own site.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works