What Are AI Crawlers Actually Seeing When They Hit Your Website?
Vercel and MERJ analyzed 500 million GPTBot requests and found zero JavaScript executions. Here's what that means for your content — and why 97% of llms.txt files aren't helping.
Vercel and MERJ went through more than 500 million GPTBot requests looking for a single instance of JavaScript execution. They didn't find one. Zero, out of half a billion requests. If your site depends on client-side rendering to populate the content that actually matters — product names, prices, FAQs, review counts — that content is simply not there when the major AI crawlers arrive.
How was this measured?
The core data here comes from three independent sources. First, Vercel and MERJ's analysis of GPTBot traffic across Vercel's hosting infrastructure — the largest published dataset on AI crawler behavior to date. Second, SearchVIU's study covering 1.3 billion AI crawler requests across 32 distinct bots. Third, Rankability's llms.txt adoption tracker, which rescans the top 1,000 domains monthly and publishes the results publicly. Where figures come from a single source, that's noted.
Do any AI crawlers actually render JavaScript?
Three. Out of the 32 crawlers SearchVIU analyzed, only Googlebot, Google's extended training crawler, and Applebot render JavaScript. The rest — GPTBot, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider, OAI-SearchBot, and others — all follow the same pattern: send a GET request, receive the raw HTML, leave.
What's odd is that these crawlers do download JavaScript files. The Vercel and MERJ data shows GPTBot fetching .js files in roughly 11.5% of its requests. ClaudeBot does it in 23.84% of requests. Neither executes a single line. They're collecting JavaScript source code as training data, not running it to build a page.
Why don't they render? Cost and architecture, mainly. A Chromium-based rendering pipeline uses significant compute per page. Googlebot has invested in that infrastructure over years. AI training crawlers are optimized for throughput at scale — they're processing hundreds of millions of pages, not running a browser for each one.
What content is actually missing?
If your app is built on a framework that does client-side rendering — and this covers a lot of default Next.js, Vue, Angular, and React configurations — what lands in the initial HTTP response may be little more than a document shell. A <div id="root"> or <div id="app">, a few <link> tags, maybe a loading spinner. Product names, prices, descriptions, structured data, and review content that define your pages populate after JavaScript runs. For AI crawlers, it never runs.
For e-commerce and product-heavy sites, this is a direct problem. SearchVIU ran a price extraction test across several AI systems to check whether they could find pricing data that only appeared via JavaScript. Google's AI system (which has the rendering pipeline) succeeded on about a quarter of tests. The others — those without rendering support — managed far fewer, with some finding nothing at all. Content that depends on a client-side API call is effectively opaque to those systems.
SearchVIU's 1.3-billion-request study identified three common failure modes. The most prevalent is a blank <body> — the crawler gets the scaffold but nothing inside it. Second is lazy-loaded content that only appears after a scroll event or IntersectionObserver fires. Third is route-level code splitting, where entire views are gated behind JavaScript module loads that never happen.
Is publishing llms.txt actually helping?
Adoption of llms.txt has grown fast — from 4,088 files tracked in June 2025 to 36,120 by May 2026, an 8.8× increase in twelve months. Rankability's latest monthly sample puts adoption at 8.7% of the top 1,000 websites, or 15.8% if you restrict to domains that were actually reachable during the scan. Shopify silently pushed the file to every merchant store on their platform in late April 2026, which moved the numbers substantially for e-commerce.
But Ahrefs analyzed 137,000 domains in May 2026 and found that 97% of llms.txt files received zero requests from AI crawlers. Of the 3% that did receive traffic, SEO audit tools and Googlebot accounted for the bulk of it. GPTBot made up 4.51% of requests to llms.txt files. The file that's supposed to guide AI crawlers is, in practice, barely being read by them.
Does this make llms.txt pointless? Not exactly. It's a useful signal for AI systems that do read it, and it costs almost nothing to publish. What it doesn't do is solve the rendering gap. If your core content is only visible after JavaScript runs, a well-crafted llms.txt pointing at those pages doesn't change what the crawler receives when it follows those links.
The Shopify example is instructive here. Shopify merchants saw high llms.txt adoption from a single platform update — but Shopify storefronts already serve substantial server-rendered HTML. Adding llms.txt on top of something that already works is a legitimate optimisation. Adding it on top of an empty shell is not.
What should you actually do about this?
Start with your server logs. Before writing any code, check what AI crawlers are actually receiving from your site. Filter your access logs for the major bot UA strings — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Applebot-Extended — and compare response body sizes against what a browser request returns. A page that's 80 KB for a human and 2 KB for GPTBot is telling you something specific. This takes about ten minutes and tells you whether you have a problem at all.
Server-side rendering is the only structural fix. If your key pages rely on client-side data fetching, SSR, SSG, or ISR for those specific paths closes the gap permanently. You don't need to convert your whole app — focus on the pages that carry commercial or informational weight: product pages, category pages, content that AI assistants might cite when recommending something to a user.
A pre-rendering layer works if you can't change the app. If a full SSR migration isn't on the table, a proxy that detects bot user-agents and serves a pre-rendered snapshot handles the gap without touching your application code. The trade-off is maintaining rendering infrastructure and keeping the cached content fresh when the underlying pages change.
The JavaScript rendering gap isn't new — traditional search engines wrestled with it for years. What's changed is that the crawlers now making consequential decisions about which content gets surfaced to AI users are specifically the ones without a rendering pipeline. That changes the stakes considerably.
Sources
- The Rise of the AI Crawler — Vercel / MERJ
- llms.txt Adoption: 8.7% of the Top 1,000 (June 2026) — Rankability
- AI Crawlers & JavaScript Rendering — SearchVIU
- llms.txt adoption rises 8.8x but 97% of files get zero AI requests — ppc.land
- We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read — Ahrefs