Guides · 2026-08-28 · By VSNARY | Emmanuel Orta · 0 views

Do AI crawlers and agents render JavaScript?

Assume not, and the question becomes much simpler: does your content exist in the response the server sends, before anything runs. Here is how to check in one request.

The safe assumption is no, and the useful reframe is that it barely matters what any individual crawler does, because the question you can actually answer is better: is your content present in the bytes the server returns, before a single line of script executes?

If yes, rendering is irrelevant — every client gets your content, executing or not. If no, you have made your visibility conditional on a behaviour you cannot verify, from vendors who do not document it, and which can change without notice.

The one-command test #

curl -s https://yoursite.com/ | sed 's/<[^>]*>/ /g' | tr -s ' ' | head -c 2000

That strips the tags from the raw response and shows you what text is actually in it. Look for your headline, your service description, your prices, your address. If what comes back is navigation and boilerplate while the substance is missing, that substance is being assembled in the browser and does not exist for a client that does not run one.

The second half of the same test is the ratio. Take the bytes on the wire against the visible text they carry. The heaviest page in this corpus delivered 476,540 bytes with 2.3% visible text. It rendered fast, it read well, no validator objected. PAGE_IS_MOSTLY_CODE fires on 27.6% of scans here, and the usual cause is inline CSS — which is also the worst kind of weight, because it cannot be cached between pages. A crawler pays for it again on every URL.

The evidence from the other direction #

There is a nice natural experiment in the challenge-page failure. A site serving a bot-verification interstitial at HTTP 200 has, in effect, put a JavaScript test on its front door: the interstitial clears itself with script after a few seconds. Browsers sail through it. In every measurement here, crawlers do not — they read the interstitial as the file. Whatever the official position of any vendor, that is what was observed on the wire.

Even where rendering does happen somewhere in a pipeline, it is a separate, later, more expensive pass. Anything that depends on it is competing for a budget that content in the first response does not have to compete for at all.

What this means for the common stacks #

That last row is the one that costs local businesses most. Hours and prices are exactly the facts assistants are asked for, and they are frequently rendered by a booking or menu widget after load — invisible to anything reading the response. Web-wide, Organization is detected on roughly 354,000 sites and OpeningHoursSpecification on roughly 23,000. Declaring you exist is common; declaring the fact somebody asked about is an order of magnitude rarer.

The second thing script hides: your URLs #

Rendering is usually discussed as a content problem. It is also a discovery problem. If your internal links exist only inside a client-side router, a reader that does not execute script sees a site with very few pages — and the fallback that is supposed to cover that is broken more often than people expect.

In this dataset 10.6% of scans find no sitemap at all, 8.8% have one that robots.txt never names, 1.2% declare a sitemap that does not resolve, and 0.7% serve an HTML page where the XML should be. A JavaScript-only navigation on top of any of those means the pages you care about are not reachable by any path a non-browser client can follow.

The check is the same one command, pointed at a link rather than a headline: fetch the raw HTML and look for real <a href> targets to your key pages. If they are generated on click, they do not exist for a client that never clicks.

Check it more than once #

Front-end deploys change what is in the first response without anyone intending to — a component moves behind a hydration boundary, a bundle splits, a widget takes over a section that used to be static. That is invisible to everyone reviewing the site in a browser.

Which is why one reading is not a state — the same panel, measured daily, reads differently across a window.

Twelve looked clean at the end and had not been clean throughout. A rendering check is a snapshot of one deploy, and the next deploy is not covered by it.

The rule that makes this a non-issue #

State every important fact twice: once in visible HTML text and once in structured data. That is not belt and braces, it is coverage of two genuinely different readers — one that parses schema, one that discards it and reads prose. A fact that exists only in JavaScript exists for neither.

Once that is true, the rendering question stops being a risk you carry and becomes trivia about someone else's pipeline.

Three findings that look alike and mean different things #

“Needs JavaScript” usually gets reported as one problem. In this corpus it is three, and the fix differs for each:

The first is fatal on any client that does not execute scripts. The second is survivable and costly. The third is the one most often missed, because it fails even for the clients that do render: a renderer that runs and cannot fetch the bundle produces the same empty page as a client that never tried. Any answer of the form “we’re fine, we server-render” is worth checking against that third row before it is believed.

We shipped three pages of our own that rendered nothing #

A worked example from this site, because it is the clearest version of this failure we have on record.

Three pages were exported from a design tool and published here. Each file looked like HTML and was not: a template with {{ }} bindings and a 69 KB client-side runtime that fetched React from a CDN and then re-read the page’s own URL to find the resources it expected. Served without the runtime, the pages shipped literal curly braces to every reader. Served with the runtime inlined into the page, they mounted and then rendered the runtime’s own source into the body, because the runtime scans the document for a marker string that its own source happens to contain.

We fixed the pages; they now mount with zero third-party scripts. The part that generalises has nothing to do with React: content that exists only after a runtime mounts is content that exists only for clients that run runtimes — and we published three of them on a site whose entire subject is that problem.

Measure the raw response, not the rendered page #

The measurements here are taken on the raw response, before anything executes. A page that reads as forty words to that measurement reads as forty words to every client that does not run your bundle — which is the entire reason for measuring it that way rather than opening the page in a browser and forming an impression.

The other direction: served less on purpose #

The mirror of a rendering problem is a delivery problem: the same URL returning different amounts of content depending on who asked. It fires on 0.5% of scans — rare, and worth knowing about because it is completely invisible from a browser.

CrawlCheck fetches the page as several named crawlers and compares the replies; a finding appears only when they differ, so no finding means delivery was uniform.

What to fix, in order #

1. Server-render or pre-render the routes that carry the facts — prices, hours, service areas, contact details.
2. Declare every one of those facts twice: once in visible HTML text, once in structured data.
3. Unblock the resources a renderer needs, or stop depending on them.
4. Then check the ratio, and keep checking it after every front-end deploy.

Done in that order, no step depends on knowing what any particular crawler does this quarter — which is the only version of this problem that stays solved.

Structured data does not rescue a page that needs JavaScript #

The advice to declare every fact twice — once in text, once in structured data — carries a condition that is easy to miss: the structured data has to be in the response too.

A JSON-LD block injected by a tag manager, a plugin that writes it on load, or a framework that emits it during hydration is subject to the identical condition as the visible text. It is not in the bytes, so it does not exist for any client that does not execute scripts. A site can pass every structured-data validator — those run a browser — and ship nothing at all to a crawler that does not.

A count of zero JSON-LD blocks in the raw response, on a page whose validator shows a clean graph, is not a contradiction; it is the same finding as the empty text, arriving through a second door.

The check is one line longer than the first one in this guide:

curl -s https://yoursite.com/ | grep -c 'application/ld+json'

Zero, on a page you believe carries schema, means the schema is being written by something the crawler never runs. Move it into the server-rendered HTML. The same applies to canonical tags, meta robots directives and hreflang: a directive that only exists after hydration is a directive most clients never receive.

Agents split the answer in two

The question gets a second answer once AI systems stop only reading and start acting. An agent working for a person reaches your site in one of two ways, and the two behave differently.

Content in the server response is the only version all three read. A browser-driving agent also meets everything a person meets, so a cookie wall or a challenge page that a human clears in a second can stop an agent halfway through a task. Check both sides: the raw response with the command above, and the page as your own connection is served it, which is what the CrawlCheck browser extension reads.

Findings behind this #

Every figure above comes from a measurement written up in full, with the date, the method and the raw numbers.

• LLMs ignored our schema and still identified us: what actually got read
• What answer engines receive from local business sites: mostly not text

Questions this post answers

Does GPTBot execute JavaScript?

Treat it as not. The dependable approach is to check whether your content exists in the raw HTML response before any script runs, because that makes the answer irrelevant. Where a JavaScript-cleared bot challenge has been served in this dataset, crawlers read the challenge page rather than clearing it.

How do I test whether my content survives without JavaScript?

Fetch the page with curl, strip the tags, and look for your headline, service description, prices and address in what remains. If the substance is missing, it is being assembled in the browser and does not exist for a client that does not run one.

My site is a single-page app. What should I change first?

Server-render or pre-render the routes that carry the facts people ask about, and declare hours, prices and contact details in structured data as well as visible text. Content injected by a widget after load is invisible to a reader that takes the first response.

Does Googlebot render JavaScript?

Yes, in a second pass after the initial fetch. That is a different question from whether an answer-engine crawler can read you, and answering it does not settle the one that matters: most crawlers feeding assistants take the first response as it arrives. Design for the first response and the rendering question stops applying to you.

How do I know if my content needs JavaScript?

Fetch the page with curl, strip the tags, and read what is left. If your headline, your prices and your service areas are in that output, no client needs to execute anything to find them. If the output is a shell and a script tag, everything below that point is visible only to clients that run your bundle.

Is a React or Vue site invisible to AI crawlers?

Not invisible. They receive whatever the server sent, which for a client-only build is the shell. The framework is not the problem; where the HTML gets assembled is. Server-rendering or pre-rendering the routes that carry your facts removes the issue entirely, and does it without depending on what any particular crawler does this quarter.

Do AI agents run JavaScript?

An agent that drives a real browser does, the same way a visitor's browser does. User-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User do not document rendering, so treat them like crawlers and put the content in the server response.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works