Is your website invisible to AI crawlers without realising it?

500 million GPTBot requests landed on pages in one network's infrastructure over the course of a single month. Zero of them executed any JavaScript. Not one. Every page where a framework assembles content client-side — where a React component fills in the product description, where a Vue router loads the article body, where an Angular service hydrates the page — GPTBot left with the same near-empty shell it got in the initial HTTP response. This is not a theoretical concern. Vercel and research firm MERJ instrumented actual server logs to measure it directly.

How was this measured?

The primary dataset here is Vercel and MERJ's network study, published December 2024, which analysed over 500 million AI crawler requests on Vercel's infrastructure. It measured which file types each crawler actually requested and whether any JavaScript was executed — not inferred from user-agent strings or robots.txt compliance, but from real network observations. Additional data comes from technology adoption surveys tracking framework usage across millions of websites and CDN network analysis covering AI crawler traffic patterns over a 12-month period. The Vercel data covers their network specifically (which skews toward modern JS frameworks); the behaviour patterns are consistent with what independent sources report.

Which crawlers actually render JavaScript?

The study produced a clean answer to a question most site owners had never thought to ask.

React adoption scales steeply with site size. Among the world's top 1,000 websites, 46.4% use React according to BuiltWith's crawl data. Among the top million sites, the figure drops to 19%. Across all websites tracked by W3Techs, it sits at 6% as of mid-2026 — up from 4.3% twelve months earlier, a 40% increase in a year. These are the sites most likely to deliver minimal content to non-JS crawlers.

The Vercel/MERJ study counted 569 million GPTBot requests, 370 million ClaudeBot requests, and millions more from PerplexityBot, Meta-ExternalAgent, and Bytespider in the study month. None of them executed JavaScript. GPTBot downloaded JavaScript files in about 11.5% of its requests — and never ran any of them. The one exception: Google-Extended, which executes JavaScript because it runs on the same infrastructure as Googlebot, which has rendered JavaScript since 2014. Everything else fetches raw HTML and stops.

If you have been checking your crawl coverage through Google Search Console, you have been measuring the one exception, not the rule.

Does the content that matters end up in the initial HTML?

For client-side rendered (CSR) sites, the answer is typically no. The initial HTTP response contains the framework bootstrap and an empty root element — a <div id="root"></div> — that JavaScript assembles into a full page after loading. The actual content: product descriptions, article bodies, pricing tables, FAQ answers, all of it only exists in the DOM after JavaScript runs. Since most AI crawlers do not run JavaScript, they read a near-empty document.

The diagnostic is simple. Fetch any important page without JavaScript:

curl -A "GPTBot/1.0" https://yoursite.com/your-page

If the response body is under a few hundred words on a page that should have thousands, you have a rendering gap. That is what 569 million GPTBot visits saw last month on those sites. That is what is going into the training datasets and search indexes built by every non-Google AI system.

The fastest-growing crawlers are exactly the ones that cannot render

CDN network data from May 2024 to May 2025 shows GPTBot's share of web crawler traffic grew 305% year over year — from 2.2% to 7.7% of all verified crawler traffic. Its real-time retrieval counterpart grew 2,825% over the same period. Googlebot — the one AI-adjacent crawler that executes JavaScript — grew 96% in the same window. Fast, but roughly three times slower than GPTBot.

The crawlers growing fastest are precisely those that cannot see your content if it requires JavaScript to render. The one crawler that can see it grew slowest.

This is a training problem, not just a search problem

The usual framing is: "will I show up in AI search results?" But that understates the problem. CDN network analysis of AI crawler request purpose finds that 80% of AI crawl volume is for model training — content ingested into a training corpus, not a queryable search index. 18% is for search indexing. About 2% is real-time retrieval driving an active user response.

Which means CSR sites are being excluded not just from AI search results today. They are being excluded from training datasets. Every time a training crawler fetches your near-empty HTML shell, your content fails to make it into the model. As training runs complete and models update, that absence compounds. The web that AI systems understand is built from the web they could actually read — and increasingly, that means only what was in the initial HTTP response.

What should you actually do about this?

Server-side rendering (SSR) and static site generation (SSG) both solve the problem at the root: they put fully-formed content in the initial HTTP response, regardless of what JavaScript does later in the browser. Both typically require meaningful rearchitecting for an existing CSR codebase, and not every team can do that immediately.

For sites where a full framework migration is not immediately feasible, pre-rendering at the edge is the practical intermediate option: detecting crawler user-agents at the edge layer and serving a pre-rendered version of each page while keeping the CSR experience intact for browser users. The pre-rendered version delivers the same content a browser would see after JavaScript hydration — product descriptions, structured data, article text — but as static HTML that any crawler can parse without executing anything.

Two approaches that will not fix this: llms.txt files (which AI crawlers almost universally never request, per separate server-log analysis of 137,000 domains) and robots.txt allow rules (which grant access but cannot help a non-JS crawler read content that does not exist in the initial HTML). The entry point is the rendered document itself.

The useful starting point is the curl test on your most important pages before committing to any solution. Product pages, documentation, pricing, landing pages — wherever content is assembled client-side, the gap is there. The fix follows the audit; the order of priority becomes clear once you have seen what the crawlers actually receive.