Findings · 2026-10-17 · By VSNARY | Emmanuel Orta

We asked an AI to pick a local contractor 422 times. The star rating decided it.

We changed one fact at a time about a fictional business and watched where the AI ranked it.

When someone asks an AI assistant “Who is the best roofer in Tulsa?”, it has to pick. What makes it pick one business over another?

We tested that directly. Not by guessing, and not by reading what AI companies say about themselves, but by changing one fact at a time and measuring what happened.

How the test works #

We gave a widely used open-weights AI model five fictional businesses of the same trade in the same city: five roofers in Tulsa, five dentists in Charlotte, five tree services in Boise. Every business came with the same list of facts: its rating, how many reviews it has, where it works, whether it is insured, its licence, its website, how long it has been in business, how fast it replies, emergency service, hours, phone and prices.

We asked the model to rank all five. Then we changed exactly one fact about one business (its rating from 4.8 to 4.1, say), kept everything else identical, and asked again. If the business fell in the ranking, that fact mattered to the model.

We ran 422 of these paired tests between October 11 and October 17, 2026, across 12 facts, 12 trades and 12 US cities.

What moved the ranking #

• Rating. Dropping from 4.8 to 4.1 stars pushed the business down in 34 of 34 tests, by 3.9 places on average. Out of five places, that is close to first-to-last.
• Number of reviews. 212 reviews versus 9: the business fell in 25 of 35 tests, 1.6 places on average.
• Serving your city. “Serves your city and nearby” versus “serves a town 40 miles away”: it fell in 24 of 36 tests, 1.4 places.
• Insurance stated. Fell in 14 of 35 tests, 0.7 places.
• A licence number shown. Fell in 18 of 36 tests, 0.7 places.
• Its own website rather than only a social media page. Fell in 13 of 34 tests, 0.5 places.

What barely mattered #

Years in business (since 2009 versus founded in 2025), reply speed, emergency service, posted hours, a listed phone number and published prices each moved the business by less than half a place. Prices mattered least of all: “call for pricing” cost almost nothing.

That surprised us. Many owners spend effort on price pages and “serving you since” badges. In this test, the model hardly noticed them. It noticed the rating, the review count and whether you work where the customer lives.

What to do with this #

• Show your real rating and review count on your homepage, as text. Not only inside a widget that loads later. If the number on your site is old, update it or connect it to your live profile.
• Name the cities you serve, in words. A map image does not say it. A sentence does.
• State insurance and your licence number plainly, if you have them.
• Do not add anything that is not true. A fact a model rewards is only worth showing if it is real. Invented reviews or numbers are the fastest way to lose a customer who checks.

The limits #

This is one model ranking fictional businesses, five at a time. It tells you what this model weighs in this setup, not what every AI does with real businesses. We are now running the identical tests on three more models from different makers, and we will publish how much they agree.

Published by CrawlCheck. CrawlCheck runs these tests, and its free scan shows which of these facts your own homepage shows, ordered by how much they moved the model.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works