GPTBot Coverage Fell from 84% to 12%. The Bot Replacing It Can't Execute JavaScript.

GPTBot's web coverage collapsed from 84% to 12% in six months. Publishers added it to robots.txt Disallow lists at scale, and the training crawler honored those rules. The crawler taking its place — OAI-SearchBot, which retrieves pages in real time when a user submits a query — reached 55.67% average web coverage by November 2025 and tripled its request volume after a major model release that month. The web is completing its transition from AI training data acquisition to AI search retrieval. There is one technical problem with that transition: OAI-SearchBot cannot execute JavaScript.

Method

Four studies provide the primary data. Hostinger's January 2026 analysis of 66.7 billion bot requests across more than five million websites, covering three separate six-day periods from June through November 2025, provides the coverage and blocking data. Botify's March 2026 analysis of over 7 billion server log files spanning November 2024 through March 2026 provides post-model-launch industry breakdowns. SearchVIU's November 2025 study, testing 32 AI crawlers across 1.3 billion requests, determines JavaScript execution capability per crawler. Vercel and MERJ's December 2024 joint analysis of over 500 million AI crawler requests on a major edge network provides request volume baselines.

The Coverage Shift: Training Down, Search Up

The Hostinger dataset tracks GPTBot coverage from 84% at the start of the June 2025 measurement window to 12% by November — a drop of 72 percentage points driven by publisher blocks. OAI-SearchBot, which powers live query-answering rather than training data collection, reached 55.67% average coverage with 279 million recorded requests over the same measurement window.

Within Botify's log file dataset, the inversion is confirmed: OAI-SearchBot now generates more crawl events per month than GPTBot. Both crawlers grew after August 2025 — GPTBot at 2.9x, OAI-SearchBot at 3.5x — but OAI-SearchBot's base had already outgrown GPTBot's declining coverage. Googlebot registered 18.2 billion events in Botify's dataset against a combined 887 million from both OAI-SearchBot and GPTBot, placing the search bot at roughly 4.9% of Google's crawl volume. That fraction is growing.

The industry-level data from Botify shows where OAI-SearchBot crawl activity increased most after August 2025.

Healthcare and media and publishing saw the largest increases — 740% and 702% respectively. These two sectors also skew heavily toward JavaScript-heavy site architectures: content management systems with client-side rendering, React-based editorial platforms, and single-page applications. The sectors seeing the biggest OAI-SearchBot growth are the same sectors with the most unresolved rendering gaps.

The JavaScript Execution Gap

SearchVIU's November 2025 analysis of 32 AI crawlers across 1.3 billion requests found that 69% of those crawlers do not execute JavaScript. The crawlers that do: Googlebot, Google-Extended, and Applebot. The crawlers that do not include GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Meta-ExternalAgent, Bytespider, and Amazonbot.

OAI-SearchBot is in the cannot-execute column — the same column as every major AI training crawler. The Vercel and MERJ analysis of over 500 million AI crawler requests confirmed zero JavaScript execution events across the entire measured dataset, which included both training and retrieval crawlers. The distinction between training and search bots is significant for robots.txt configuration; it is irrelevant to JavaScript rendering. Both categories receive whatever is present in the initial HTTP response body and nothing more.

For a site built with client-side-only rendering, that initial response body contains an HTML scaffold with a root container element, script tags, and no page content. For a site with deferred data fetching — server-side rendering that generates layout but loads product listings, article bodies, or pricing data from a separate API call after initial page load — the crawler receives the layout structure with empty content slots.

The diagnostic for either failure mode is a plain HTTP GET against any representative URL. If the response body is missing the page's primary text, the page is invisible to 69% of AI crawlers, including OAI-SearchBot.

Three Failure Modes in Practice

Blank body: Pure client-side rendering. The server returns only an HTML scaffold — typically a root container element and script tags. All content is injected after JavaScript mounts. AI search crawlers receive an empty document and index nothing useful.

Deferred content: The server returns HTML with navigation and layout, but primary content — product listings, article bodies, pricing data — is loaded from a separate API call after render. The crawler indexes the layout without the payload. The dominant search engine, which does execute JavaScript, has documented indexing failures on product pages where dynamically loaded items were partially or entirely missed; AI search bots that skip JavaScript execution receive none of those items.

Hydration mismatch: Server-rendered HTML that differs from the hydrated client state. Slow partial hydration means components missing from the initial server-rendered pass never appear in the crawler's snapshot. A crawler that indexes the initial server response may index a page version that does not match what human users see — or that is missing the page's main substance entirely.

What This Means for Site Owners

OAI-SearchBot's 702% increase in media and publishing requests is not a future trend to prepare for. It represents current crawl volume hitting sites right now. A site serving rendered HTML on those requests gets indexed; a site serving a JavaScript scaffold gets indexed as an empty page. The gap is deterministic and measurable with a single curl request.

Separating robots.txt rules by crawler purpose is the correct first step. A Disallow on GPTBot blocks training data collection. A Disallow on OAI-SearchBot blocks citation eligibility. Most robots.txt configurations written in 2023 or 2024 were drafted before OAI-SearchBot existed as a distinct user-agent. They are worth reviewing against the current bot registry.

For JavaScript-heavy sites, dynamic rendering — detecting AI search crawler user-agents and serving pre-rendered HTML to them while human visitors receive the client-side application — is the incremental path that avoids a full migration to server-side rendering. OAI-SearchBot, PerplexityBot, and Meta-ExternalAgent all declare their identity via documented user-agent strings with published IP ranges available for verification. The routing logic can be added at the edge without changing the application codebase.

The curl test on your highest-value pages is the starting measurement. If the response body is missing product descriptions, article text, or structured data that appears in a browser, that content gap is what OAI-SearchBot indexes on every crawl.