GPTBot Downloads JavaScript on 11.5% of Crawls. It Executes Zero of Them.

GPTBot fetches JavaScript files on 11.5% of its logged crawl requests. Execution events in that same dataset: zero. ClaudeBot retrieves JavaScript files at more than double that rate—23.84%—with identical execution output: none. Across a 500M+ request corpus analyzed by Vercel and MERJ in their 2024–2025 AI crawler study, not a single AI bot in the static-tier cohort executed JavaScript. If your site relies on client-side rendering for any meaningful portion of its content, the majority of AI crawlers cannot see it.

Method

This analysis synthesizes three sources: the Vercel/MERJ AI crawler request corpus (500M+ logged fetches from GPTBot and ClaudeBot), a separate analysis by searchVIU of 1.3 billion AI crawler requests across 23 distinct bots (November 2025), and public documentation from search engine engineering blogs. The Vercel/MERJ dataset was collected via edge request logs across a large distributed sample of production sites. The searchVIU dataset draws from crawler-facing telemetry across a European web sample. Neither dataset requires inference; both record observed fetches directly.

Three Rendering Tiers

Not all AI crawlers are equivalent. Observed behavior segments them into three tiers based on their documented and empirically confirmed capability to execute JavaScript.

Tier 1: Full Web Rendering Service

Google operates a centralized Web Rendering Service (WRS) used by both its traditional search crawler and its AI search crawler, Google-Extended. WRS executes JavaScript, handles Shadow DOM, and can trigger lazy-load events. A Google engineering lead confirmed in 2024 that Google-Extended draws on the same rendering infrastructure as Googlebot. For these crawlers, client-side rendered content is visible—assuming crawl budget is allocated to the URL and the content renders within the time window.

Bingbot occupies a partial Tier 1 position. It uses a headless browser runtime internally but deploys it selectively. Rendering is less reliable and less consistent across sites than the WRS model, and Bingbot's AI search derivative inherits these limitations.

Tier 2: Fetch-Only with Header Negotiation

This tier fetches resources and follows redirects but does not execute JavaScript. GPTBot and OAI-SearchBot fall here. These crawlers fetch HTML, images, and frequently JavaScript files—but the JavaScript is fetched for analysis purposes (link extraction, asset dependency mapping) rather than execution. The page state they receive is equivalent to a freshly loaded document with no DOM mutations applied.

ClaudeBot shows a higher JavaScript fetch rate than GPTBot (23.84% vs 11.5%), suggesting more aggressive asset graph traversal, but the outcome is identical: zero execution across the full dataset.

Both bots perform HTTP header negotiation and follow robots.txt directives. X-Robots-Tag headers are respected. For sites with JavaScript-dependent content, neither bot can see what users see.

Tier 3: HTML Document Request Only

PerplexityBot, Meta-ExternalAgent, Bytespider, and the majority of smaller AI research crawlers operate in this tier. They issue a single HTTP GET request for the HTML document at the target URL and nothing more. No sub-resource fetches, no header negotiation beyond the initial request. What is returned in the initial HTML response body is the entirety of what they receive.

For JavaScript-heavy single-page applications, these crawlers receive a near-empty shell—typically a root container element and script tags pointing to bundles they will never fetch.

JavaScript Capability Across the AI Crawler Landscape

The searchVIU November 2025 dataset puts the scale of this problem in context. Across 23 distinct AI crawler bots and 1.3 billion analyzed requests, 69% of bots had no JavaScript execution capability whatsoever. The 31% that did is heavily weighted toward Google's indexing infrastructure.

This distribution has direct consequences for content visibility. A page that requires JavaScript to render its primary content—product listings, article bodies, pricing tables—is invisible to the majority of AI crawlers making indexing and retrieval decisions today.

Diagnosing Your Exposure

Two fetch patterns reliably distinguish Tier 1 from Tier 3 crawlers in your own server logs. First: examine the content-type distribution of requests from each bot UA. GPTBot requests are 57.7% HTML documents and 11.5% JavaScript files; ClaudeBot requests are 35.17% images. If your logs show a bot requesting your JavaScript bundle files but no subsequent state changes appear in the page structure, it is analyzing your asset graph, not rendering your pages.

Second: serve a minimal server-rendered HTML response with no external script dependencies on a test URL, then serve the same logical content via client-side rendering on a parallel URL. If both URLs appear in the bot's fetch log but the CSR variant shows no sub-resource fetches, the bot did not render. The differential in your analytics will confirm which tier that crawler belongs to.

What This Means for Site Owners

Sites built on server-side rendering or static-site generation have nothing to configure here. The HTML response body is the content the crawler receives; rendering tier is irrelevant because the content is already present in the initial response.

For sites using client-side rendering—frameworks where the initial HTML is a minimal shell populated by JavaScript—the exposure depends on which crawlers matter for your traffic mix. If AI-generated search citations are a meaningful channel, Tier 3 coverage is structurally poor. Dynamic rendering (serving pre-rendered HTML to known bot UAs while serving the SPA to human browsers) is the established remediation. It does not require rebuilding the application; it requires intercepting requests from bot user-agent strings at the edge and returning pre-rendered output.

For sites already returning server-rendered HTML for Googlebot via an existing pre-render layer, that layer will not automatically cover AI crawlers unless those UAs are added to the interception rules. GPTBot, ClaudeBot, PerplexityBot, and the user-agent strings for other major AI products each need to be included explicitly—they will not match Googlebot patterns.

One scenario where rendering tier is not the constraint: AI search products that retrieve content at query time rather than through crawl indexing operate differently. These systems make live fetch requests when a user query matches a candidate URL. If those retrieval fetches use a headless browser runtime, the rendering tier of the background indexing crawler becomes secondary. Verify retrieval behavior per product before attributing ranking gaps to rendering alone.