Findings · 2026-10-09 · By VSNARY | Emmanuel Orta · 6 views
What AI crawlers get from Wix, Shopify, Squarespace and Webflow sites: 980 sites measured
We fetched 980 homepages on four website builders as GPTBot, ClaudeBot and PerplexityBot. The platform decides most of the answer: Shopify and Wix publish an llms.txt for nearly every site, Wix serves crawlers a lighter page, and Squarespace’s AI switch leaves ChatGPT search and Perplexity open.
On 9 October 2026, across 980 homepages in the Tranco top 200,000, AI crawlers were almost never refused by the platforms themselves. Shopify publishes a templated llms.txt on 88% of stores (98% serve one), Wix generates one on 96% of sites and serves AI crawlers a lighter HTML version with the same visible text, Squarespace sites rarely have an llms.txt (1%) and 8% use its AI crawler block, which does not cover ChatGPT search or Perplexity, and Webflow results depend on the owner (41% llms.txt, 60% JSON-LD).
Disclosure: CrawlCheck publishes this study and sells a check for the same things it measures. Platform features are described from each platform’s own help pages, linked below. No individual site is named.
On a hosted website builder, most of what an AI crawler receives is decided by the platform, not the site owner: the robots.txt, whether an llms.txt exists, which structured data the homepage carries and, on one platform, a different version of the page itself. We measured what 980 sites on Wix, Shopify, Squarespace and Webflow served to GPTBot, ClaudeBot and PerplexityBot on 9 October 2026. Three of the four platforms produce nearly the same result on every site. The differences that matter are between platforms, and in a handful of owner settings.
How we measured #
We took the top 200,000 domains of the Tranco research ranking (list PY35J), found the sites hosted on each platform and confirmed the platform from the homepage itself. That gave 2,678 Shopify candidates, 1,168 Webflow, 190 Squarespace and 156 Wix. We measured up to 400 of the highest-ranked confirmed sites per platform, one homepage per site after following redirects: fetched as a desktop browser and as each AI crawler, with its robots.txt read for each crawler under the standard rules (RFC 9309), its /llms.txt, and the JSON-LD on the page. A fetch that returned 429 or 5xx was retried once after four seconds. Squarespace sites rate-limited our run with 429 even after the retry; we rechecked by hand that those sites serve normally, and 429 is not counted as a refusal anywhere below.
The four platforms side by side #
| Shopify | Wix | Square | Webflow | |
|---|---|---|---|---|
| Sites measured | 399 | 112 | 74 | 395 |
| Median Tranco rank | 49k | 110k | 75k | 38k |
| robots.txt closes at least one AI crawler | 1.0% | 0% | 8.1% | 3.8% |
| Homepage refused to GPTBot and ClaudeBot (403) | 0.8% | 0% | 1.4% | 1.3% |
| AI crawlers served a different HTML version | 0.3% | 94.6% | 2.7% | 0% |
| AI crawlers got under half the visible text | 0% | 0.9% | 0% | 0% |
| Serves llms.txt | 97.7% | 96.4% | 1.4% | 41.3% |
| JSON-LD on the homepage | 86.7% | 95.5% | 94.6% | 59.5% |
| Organization or LocalBusiness markup | 76.7% | 39.3% | 64.9% | 46.8% |
| robots.txt points to a sitemap | 98.7% | 98.2% | 83.8% | 89.1% |
Shopify: an llms.txt on nearly every store, written by Shopify #
390 of 399 Shopify homepages (97.7%) serve an llms.txt. 350 of them (87.7%) are the same templated file, headed “Agent Instructions” with the store’s name, served as text/markdown: guidance for shopping agents on how to browse and buy from the store. The other 40 stores replaced it with their own. robots.txt points to a sitemap on 98.7% of stores, and 76.7% carry Organization or LocalBusiness markup on the homepage.
The refusals are not Shopify’s. Three stores answered GPTBot and ClaudeBot with a 403 “Your request was blocked.” page from bot protection in front of the store, while letting PerplexityBot and a browser through. On all three, robots.txt allowed both crawlers: the file says yes and the server says no. A crawler that obeys robots.txt still gets nothing, and the owner who reads robots.txt to check believes the site is open.
Wix: a separate, lighter page for AI crawlers, with the same text #
Wix serves known crawlers a different HTML version of the page. On 106 of 112 sites (94.6%) the HTML GPTBot received differed in size by more than a fifth from the browser’s: a median of 0.68 MB against 1.16 MB. The visible text matched within 2% on 106 of 110 sites, so for almost every site the lighter version carries the same words with less code. One site was the exception: the crawlers received 2,872 characters of visible text against the browser’s 8,024, including site styling code that leaked into the text. We fetched it again by hand, twice, with the same result.
Wix also generates an llms.txt for each site (108 of 112, 96.4%), which the owner can edit or switch off in the dashboard, according to Wix. JSON-LD is on 95.5% of Wix homepages, but only 39.3% of them describe the business as an Organization or LocalBusiness, the lowest of the four platforms. None of the 112 sites closed an AI crawler in robots.txt.
Squarespace: almost no llms.txt, and an AI switch that is mostly a training opt-out #
One Squarespace site in 74 serves an llms.txt. JSON-LD is on 94.6% of homepages and 64.9% describe the business. robots.txt points to a sitemap on 83.8%, the lowest of the four.
Six sites (8.1%) have turned on Squarespace’s option to block AI crawlers, found in Settings under Crawlers. We read the list it writes into robots.txt. It closes GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot, DuckAssistBot and eighteen others. It does not list OAI-SearchBot or PerplexityBot, the crawlers behind ChatGPT search and Perplexity answers, so a site with only this option on stays readable to them. Four of the six also have the search engine option on, which closes the whole site to every crawler, Googlebot included.
One Squarespace homepage refused all three AI crawlers with a plain 403 from a proxy in front of the site, again with a robots.txt that allowed them.
Webflow: the owner decides, and the results show it #
Webflow ships less by default, so its numbers spread out. 41.3% of sites serve an llms.txt, which on Webflow is a file the owner uploads (Webflow help). 59.5% have JSON-LD on the homepage, the lowest of the four, and 10 sites (2.5%) carry a JSON-LD block that does not parse. 15 sites (3.8%) close at least one AI crawler in their own robots.txt, the most of any platform where the owner writes the file. Five sites block GPTBot and ClaudeBot at the server while their robots.txt allows them, the same contradiction as on Shopify.
What to check on your own site #
- Shopify: read the llms.txt Shopify publishes for your store and decide whether those are the instructions you want agents to follow. If you use bot protection, make sure it agrees with your robots.txt. Shopify steps
- Wix: compare what a crawler receives with what a browser shows; on almost every site they match, but check yours. Add Organization or LocalBusiness details so the business is described, not only the page. Wix steps
- Squarespace: if you turned on the AI crawler option, know that it stops training crawlers and some assistants but not ChatGPT search or Perplexity, and that the search engine option next to it closes the site to everyone. Squarespace steps
- Webflow: add JSON-LD for the business and an llms.txt if you want one; neither exists until you add it. Webflow steps
The free CrawlCheck audit fetches your homepage as these crawlers and shows the robots.txt verdict, what each one received, the llms.txt and the structured data, in one report.
Every figure above came out of CrawlCheck.
Point it at your own domain and see the same measurements, free.
The main product
Found this on your own site? We fix it for $749.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Questions this post answers
Do Wix, Shopify, Squarespace or Webflow block AI crawlers?
Not by default. In our 9 October 2026 measurement of 980 homepages, refusals came from the sites’ own bot protection or from options the owner turned on: 0.8% of Shopify stores, 1.3% of Webflow sites and 1.4% of Squarespace sites refused GPTBot and ClaudeBot, and no Wix site did.
Does Shopify create an llms.txt for my store?
Yes for most stores we measured: 97.7% of Shopify homepages served an llms.txt and 87.7% were the same templated “Agent Instructions” file naming the store.
Does Squarespace’s AI crawler setting block ChatGPT search?
No. The list it writes into robots.txt closes GPTBot, ClaudeBot, Google-Extended, CCBot and more than twenty other crawlers, but not OAI-SearchBot or PerplexityBot. The separate search engine option closes the site to every crawler.
Why does Wix serve AI crawlers a different page?
Wix serves known crawlers a lighter HTML version (a median of 0.68 MB against 1.16 MB in our sample). On 106 of 110 sites the visible text matched within 2%, so the content is the same with less code.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.