CrawlCheck

Guides · 2026-08-28 · By

Do AI crawlers render JavaScript?

Assume not, and the question becomes much simpler: does your content exist in the response the server sends, before anything runs. Here is how to check in one request.

The safe assumption is no, and the useful reframe is that it barely matters what any individual crawler does, because the question you can actually answer is better: is your content present in the bytes the server returns, before a single line of script executes?

If yes, rendering is irrelevant — every client gets your content, executing or not. If no, you have made your visibility conditional on a behaviour you cannot verify, from vendors who do not document it, and which can change without notice.

The one-command test

curl -s https://yoursite.com/ | sed 's/<[^>]*>/ /g' | tr -s ' ' | head -c 2000

That strips the tags from the raw response and shows you what text is actually in it. Look for your headline, your service description, your prices, your address. If what comes back is navigation and boilerplate while the substance is missing, that substance is being assembled in the browser and does not exist for a client that does not run one.

The second half of the same test is the ratio. Take the bytes on the wire against the visible text they carry. The heaviest page in this corpus delivered 476,540 bytes with 2.3% visible text. It rendered fast, it read well, no validator objected. PAGE_IS_MOSTLY_CODE fires on 13.7% of scans here, and the usual cause is inline CSS — which is also the worst kind of weight, because it cannot be cached between pages. A crawler pays for it again on every URL.

The evidence from the other direction

There is a nice natural experiment in the challenge-page failure. A site serving a bot-verification interstitial at HTTP 200 has, in effect, put a JavaScript test on its front door: the interstitial clears itself with script after a few seconds. Browsers sail through it. In every measurement here, crawlers do not — they read the interstitial as the file. Whatever the official position of any vendor, that is what was observed on the wire.

Even where rendering does happen somewhere in a pipeline, it is a separate, later, more expensive pass. Anything that depends on it is competing for a budget that content in the first response does not have to compete for at all.

What this means for the common stacks

PatternSurvives without JS?What to do
Server-rendered pagesYesNothing; check the ratio
Static site generatorsYesNothing; check the ratio
Client-only SPANoServer-render or pre-render the routes that matter
Content in a JS widget or tabUsually noPut the text in the HTML; hide with CSS, not script
Prices or hours injected by an appNoAlso declare them in structured data

That last row is the one that costs local businesses most. Hours and prices are exactly the facts assistants are asked for, and they are frequently rendered by a booking or menu widget after load — invisible to anything reading the response. Web-wide, Organization is detected on roughly 354,000 sites and OpeningHoursSpecification on roughly 23,000. Declaring you exist is common; declaring the fact somebody asked about is an order of magnitude rarer.

The second thing script hides: your URLs

Rendering is usually discussed as a content problem. It is also a discovery problem. If your internal links exist only inside a client-side router, a reader that does not execute script sees a site with very few pages — and the fallback that is supposed to cover that is broken more often than people expect.

In this dataset 5.6% of scans find no sitemap at all, 3.2% have one that robots.txt never names, 2.3% declare a sitemap that does not resolve, and 1.5% serve an HTML page where the XML should be. A JavaScript-only navigation on top of any of those means the pages you care about are not reachable by any path a non-browser client can follow.

The check is the same one command, pointed at a link rather than a headline: fetch the raw HTML and look for real <a href> targets to your key pages. If they are generated on click, they do not exist for a client that never clicks.

Check it more than once

Front-end deploys change what is in the first response without anyone intending to — a component moves behind a hydration boundary, a bundle splits, a widget takes over a section that used to be static. That is invisible to everyone reviewing the site in a browser.

Across 48 sites measured every day for fifteen days here, 68.8% carried a defect on the final day and 93.8% carried one at least once in the window. Twelve looked clean at the end and had not been clean throughout. A rendering check is a snapshot of one deploy, and the next deploy is not covered by it.

The rule that makes this a non-issue

State every important fact twice: once in visible HTML text and once in structured data. That is not belt and braces, it is coverage of two genuinely different readers — one that parses schema, one that discards it and reads prose. A fact that exists only in JavaScript exists for neither.

Once that is true, the rendering question stops being a risk you carry and becomes trivia about someone else's pipeline.

Every figure above came out of this scanner.

Point it at your own domain and see the same measurements, free.

Scan a domain — free

Questions this post answers

Does GPTBot execute JavaScript?

Treat it as not. The dependable approach is to check whether your content exists in the raw HTML response before any script runs, because that makes the answer irrelevant. Where a JavaScript-cleared bot challenge has been served in this dataset, crawlers read the challenge page rather than clearing it.

How do I test whether my content survives without JavaScript?

Fetch the page with curl, strip the tags, and look for your headline, service description, prices and address in what remains. If the substance is missing, it is being assembled in the browser and does not exist for a client that does not run one.

My site is a single-page app. What should I change first?

Server-render or pre-render the routes that carry the facts people ask about, and declare hours, prices and contact details in structured data as well as visible text. Content injected by a widget after load is invisible to a reader that takes the first response.

Related findings

All guides · The dataset · How the dataset works