Why Is Your Site Invisible to AI Search? The JavaScript Rendering Problem Nobody's Talking About

A Vercel analysis of over 500 million AI crawler requests found not a single confirmed instance of JavaScript execution. Not one. If your website is built on a client-side rendered framework — and statistically there's a solid chance it is — the AI search engines that your potential customers are increasingly relying on might be seeing nothing but an empty shell where all your content should be.

So what's actually going on here, and how wide is the gap between what users see and what AI crawlers get?

How did we find this out?

In late 2024, Vercel partnered with SEO research firm MERJ to analyse crawler behaviour across their hosting network. They logged over 500 million GPTBot requests and looked at what types of content the bot was fetching — and, critically, what it did with JavaScript files once it had them.

The answer turned out to be: nothing useful. GPTBot downloads .js files in about 11.5% of its requests, but those files are retrieved as raw text (likely for training data purposes) and never actually run. The bot reads the initial HTML response that the server sends back, and that is the entire extent of its content processing. Whatever gets injected into the DOM after the page loads is completely invisible.

The same pattern holds for the other major AI crawlers. ClaudeBot fetches JavaScript at an even higher rate — 23.84% of requests — but equally never executes any of it. No dynamic content, no lazy-loaded product listings, no client-rendered article text.

There is one notable exception: the AI search crawler that inherited its rendering infrastructure from the world's most widely-used traditional search engine. That crawler genuinely does execute JavaScript, which gives it a fundamentally different view of modern web applications compared to every other AI crawler out there. But it is the only one. If you're assuming that being visible to it means you're visible everywhere in AI search, you're working with an incomplete picture.

What does a non-rendering crawler actually see on your pages?

This depends almost entirely on your rendering architecture — and this is where things get genuinely uncomfortable for a large chunk of the web.

For server-rendered pages — proper server-side rendering, static site generation, or hybrid approaches — the initial HTML response contains most of the meaningful content. An AI crawler reading only initial HTML still gets the substance of your pages. The problem is concentrated almost entirely in client-side rendered single-page applications.

On a typical React SPA built with client-side rendering, the initial HTML is essentially a scaffold: a <div id="root"></div> placeholder, some stylesheet links, some script tags, and a handful of meta tags. Your actual content — the product listings, the article text, the FAQ sections, the pricing tables, the structured data — all of that gets injected by JavaScript after the page loads. A crawler that never runs that JavaScript walks away thinking your page is near-empty.

Research into real CSR pages has found that 50%–80% of meaningful content on these sites is invisible to non-rendering crawlers. That is not an edge case or a minor SEO optimisation gap — that is most of your content, unreachable by the systems increasingly driving how people discover products and services.

Here is the scale of the exposure: React now powers over 46% of the world's top 1,000 websites. And despite the engineering community conversation pushing everyone toward server-side rendering, client-side single-page applications still represent around 84% of React implementations in active use. A very large percentage of the modern web is built in exactly the way that makes it invisible to AI crawlers.

Is llms.txt going to solve this?

There has been a lot of discussion about the llms.txt specification — a file you place at your domain root that gives AI systems a structured guide to your content and how it should be used. It's a reasonable concept, but the current adoption numbers paint a more complicated picture than the hype suggests.

As of mid-2026, just 5.61% of the top 10,000 websites have a valid llms.txt file. And interestingly, adoption doesn't correlate with website size or traffic volume — high-traffic sites with over 100,000 monthly visits actually show slightly lower adoption at 8.27% than mid-tier sites at 10.54%.

The headline adoption figure you may have seen — that 78% of Shopify stores now have llms.txt — is technically accurate but needs some context. That number reflects an automatic platform rollout pushed by the e-commerce platform itself in April and May 2026, not a deliberate decision by merchants. The spec appeared on 78% of those stores because the platform put it there by default.

More fundamentally, llms.txt does not solve the rendering problem. An AI crawler cannot read content that only exists in the rendered DOM, regardless of what your manifest file says. The file might tell a crawler what sections your site contains and how you'd like your content to be used — but if the crawler fetches those URLs and gets CSR scaffolding back, the manifest helps very little.

There is also no confirmed statistical correlation yet between having a llms.txt and receiving more AI citations. Early data from studies tracking this show no measurable effect on how frequently a domain gets cited in AI answers.

What is actually worth doing about this?

The most complete fix is to get your meaningful content into the initial HTML response. For React SPAs, that means adopting server-side rendering or static site generation. Modern frameworks have made this substantially more achievable than it was a few years ago, but it is still a real migration project that requires engineering time and testing.

For teams where a full rendering migration isn't on the near-term roadmap, pre-rendering is a practical middle ground. You detect AI crawler user-agent strings at the edge and serve a fully rendered HTML snapshot while regular users continue to get the interactive client-side experience. The snapshots need to be kept reasonably fresh, but the crawlers get content-rich HTML rather than an empty scaffold.

Whatever you decide to do architecturally, start by actually checking your server logs for AI crawler activity. Look for GPTBot, ClaudeBot, and PerplexityBot requests and compare the response sizes against what a real browser session returns. If you're serving a few hundred bytes to crawlers that should be receiving twenty-plus kilobytes of content, that's a fairly clear signal they are getting shells rather than pages.

The JavaScript rendering gap has been a concern for traditional search for years, and the broader web ecosystem has been moving toward server-first rendering as a default. AI crawlers are a few years behind that curve. Most have not yet built the infrastructure to render JavaScript at the scale they crawl. The sites that are proactively getting rendered content in front of those crawlers right now are the ones that will show up in AI answers while their competitors stay dark.