AI crawlers don't run your JavaScript

In September we checked whether the AI crawlers could reach this site. GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot all got a 200 on every page we tested, and for about a day I thought that settled it.

Then the obvious objection came up: a 200 only means the file was delivered. It doesn't say whether anything inside it was read, and on a site built the way most sites are built now, the content you care about isn't in that file in the first place.

Which crawlers actually render JavaScript?#

Vercel and MERJ watched live crawler traffic on production sites, and none of the major AI crawlers executed JavaScript. The crawlers from OpenAI, Anthropic, Perplexity, Meta and ByteDance all download the .js files - so the request shows up in your logs and you pay for the bandwidth - and then never run them.

They download quite a lot of them, too. In that sample, 11.50% of ChatGPT's crawler requests went to JavaScript files, and 23.84% of Claude's. That's a big share of the crawl spent on files whose contents nobody reads.

Google and Apple are the exceptions:

Crawler	Renders JavaScript	Why
Googlebot	Yes	Has rendered for years
Google Gemini	Yes	Rides Googlebot's infrastructure
Applebot	Yes	Renders for Siri and Spotlight
GPTBot, OAI-SearchBot	No	Fetches scripts, does not execute
ClaudeBot	No	Same
PerplexityBot	No	Same
Bytespider, Meta-ExternalAgent	No	Same

If your site is a single-page application, look at that middle column closely. The same page can be fully visible to Google's AI features and empty for ChatGPT and Perplexity, and you won't get an error anywhere.

One thing to keep in mind about the research: it ran in December 2024, and crawler behaviour can change. Rendering costs a lot of compute, but a lab with enough money can decide to pay for it. So I'd treat the table as the best measurement we have right now, and re-run the check on your own pages instead of trusting a table in a blog post, including this one.

What does "doesn't render" mean for a page that works fine in my browser?#

It means the crawler reads your HTML file before any JavaScript has touched it.

For a server-rendered page, that file already has almost everything in it: the headings, paragraphs, tables and links. For a client-rendered page it can be close to empty - a <div id="root">, a few script tags and a loading state. Your browser builds the real page out of that in a few hundred milliseconds, which is why this is so easy to miss when you check your own site the way everyone does, in a browser.

Nothing else will show it to you either. Your analytics don't, because the crawler never ran the analytics script, and your server logs show a successful request. The only sign is that you aren't being quoted and can't work out why.

How do I see what a crawler sees?#

Fetch the page without a browser and read what comes back:

curl -s https://example.com/your-page | head -100

That's roughly what a crawler that doesn't render gets. If your headline, your first paragraph and your main links are in there, you're fine. If you get back an empty shell and a script tag, that's all GPTBot has to work with.

Two more checks are worth doing at the same time. Open the page in your browser with JavaScript turned off, which catches content that only loads when you scroll or click. And make sure the crawlers aren't blocked completely - plenty of sites block them with a CDN rule nobody remembers turning on, and fixing the HTML won't help with that.

Why is "we rank fine on Google" no longer reassurance?#

Because Google is the one engine that renders JavaScript anyway.

Googlebot has done it for years and Gemini uses the same setup. So a client-rendered site can rank well and pass every SEO audit while the engines that don't render see nothing. Most tools don't check for this, because for about ten years there was no reason to.

It adds to something we wrote about in our piece on GEO and SEO: the engines don't cite the same sources anyway. Bing Chat and Perplexity overlap on roughly 26% of the domains they cite. If your content is also invisible to the half that doesn't render, working on your rankings won't change anything for those engines, because your content never reached them.

What this changed on our own site#

Two things, and we found the second one by getting it wrong.

When we built this blog we went with plain Markdown rendered at build time instead of MDX, so that nothing in a post depends on JavaScript running. A chart that only appears after hydration can't be quoted, so any figure we publish gets rendered into the HTML during the build.

The second one was our analytics. In September they recorded nothing for about two weeks. This site is served as static assets from a Cloudflare Worker, and the analytics script was supposed to be injected automatically by a step that only runs on proxied origin responses, which ours never are. So the script simply wasn't in the HTML, and the dashboards looked believable the whole time.

That's the same lesson again from a different side. Only what's in the delivered HTML is really there, and the way to know what's in it is to fetch the page and read it.

Is there something between a rewrite and doing nothing?#

Yes, and for most sites it's the sensible option.

Prerendering sits in between. People still get the client-rendered app, and a build step or an edge service makes a plain HTML version for anything that doesn't run scripts. Frameworks have different names for it (static generation, server-side rendering, a prerender step), and which one you use matters much less than whether any of them is switched on.

Prerendering has two risks. Serving crawlers different HTML than people, based on detecting who's asking, has gone wrong for a lot of sites over the years and is rarely recommended. The safe version is the same HTML for everyone, with the browser adding to it. And a prerender service is one more thing that can break without anyone noticing - which would leave you with exactly the problem this post is about, a page that looks fine to you and is empty for everything else.

What should you actually check?#

Four things, starting with the one that costs you the most.

Can the crawlers reach you at all? A CDN rule or a line in robots.txt can make everything else on this list irrelevant, and one request per crawler is enough to find out.

What's in the HTML before JavaScript runs? That's the curl from above. If you get a shell back, that's your biggest problem, and better writing won't fix it.

Are your key pages server-rendered? Most frameworks can do it, and a lot of projects just never turned it on. The pages that matter are the ones you'd want quoted, like your service pages and your best writing, and your dashboard can stay as it is.

Is fixing it worth it for you? If your business doesn't depend on being found through an AI answer, a client-rendered app is completely fine to have - a booking tool for people who already have the link doesn't need to be quotable. What matters is knowing which case you're in now instead of finding out next year.

None of this means you should rebuild a site that works. The pages that need to arrive as HTML are usually a small set you can name: your service pages, your articles, whatever someone would ask an assistant about. Most frameworks will render those at build time if you change a setting.

Frequently asked questions#

Does this mean I should not use React or Vue?#

No, keep them! The pages you want quoted just need to arrive as HTML, and every major framework supports server rendering or static generation. What matters is whether your project uses it for those pages.

If a crawler downloads my JavaScript file, doesn't it read the content inside?#

It downloads the file and that's it. Running the code is a separate step that costs a lot more, and the Vercel and MERJ research found that the major AI crawlers skip it. Anything that only appears after the code runs, they never see.

My site ranks well on Google. Am I affected?#

Possibly, and Google won't tell you. Googlebot renders JavaScript and Gemini uses the same setup, so Google is the one place where this problem doesn't show up. You have to test the engines that don't render separately.

How is this different from normal SEO advice about JavaScript?#

The advice is the same, but more depends on it now. For years you could count on Googlebot rendering your page at some point. A crawler that never renders will never see that content at all.

We use Next.js or Nuxt. Are we fine by default?#

Usually yes for pages that are statically generated or server-rendered, and no for pages that load their content in the browser after the page has loaded. Most projects have both, which is why you run the curl check per page and not per framework.

Is there a tool that checks this for me?#

There are several, but curl answers the main question in one line and isn't trying to sell you anything. A tool makes sense once you need to check hundreds of pages.

Does server rendering help with Google's AI Overviews too?#

It doesn't change whether Google can see the page, because Googlebot renders either way. It does shorten the time between publishing and being crawlable, which matters more for news than for articles that stay relevant for years.

Does an llms.txt file solve this?#

No. Google's guidance on its AI features says you "don't need to create new machine readable files, AI text files, or markup" to show up in them. And a file that lists your content doesn't help if the content is missing from the HTML the crawler actually fetched.

How often should I re-check?#

Once a quarter is enough for most sites, plus after any change to your framework or hosting. The second one matters more - our own analytics broke because of a change in how the site was delivered, and nobody had edited a single page.

---

Published 7 October 2026. Sources: research by Vercel and MERJ on AI crawler behaviour, Google Search Central's guidance on AI features, and our own crawler-reachability checks on this domain in September 2026.