OAI-SearchBot Overtook GPTBot: The Inversion Site Owners Missed

After August 2025, a shift in AI crawler logs happened that most site operators have not noticed. OAI-SearchBot — the real-time retrieval bot powering live AI search citations — overtook GPTBot in raw server-log volume across enterprise sites. Botify's analysis of more than 7 billion bot log events spanning November 2024 through March 2026 confirms it: OAI-SearchBot recorded 3.5× more events in the post-August period than before; GPTBot recorded 2.9×. That gap — in favour of the search crawler — is widening. If your robots.txt treats both bots identically, you may be blocking the bot that drives live AI citations without realising it.

Method

Two datasets underpin this analysis. First, Botify's enterprise log file dataset: more than 7 billion bot events across clients in retail, media, healthcare, software, travel, and marketplaces, covering November 2024 through March 2026. Second, a public crawl report from a major internet infrastructure provider, covering October–November 2025, which measured what fraction of unique web pages each crawler reached across tens of millions of network zones. The datasets are complementary: Botify captures request frequency and volume per site; the coverage report captures how broadly each bot spans the web.

The Volume Inversion

Before the new flagship model launch in August 2025, GPTBot dominated crawl footprints in Botify's enterprise logs. After that point, OAI-SearchBot crossed above it. In absolute terms, OAI-SearchBot added 2.2 billion events to Botify's dataset; GPTBot added 1.8 billion. The two bots serve entirely different functions: GPTBot collects training data (batch, periodic, depth-first traversal of the link graph); OAI-SearchBot indexes content for live search citations (near-real-time, breadth-first, skewed toward high-authority and freshness-relevant pages). A site that disallows GPTBot but allows OAI-SearchBot can still appear in live AI search results; the reverse is not true.

The coverage report adds context on breadth. In October–November 2025, Googlebot reached 11.6% of unique web pages in the measured sample. GPTBot reached 3.6%. ClaudeBot and Meta-ExternalAgent each reached 2.4%. PerplexityBot reached just 0.06% — approximately 200× fewer unique pages than Googlebot — consistent with an on-demand pattern where it fetches pages as user queries arrive, rather than proactively indexing the long tail.

Industry Concentration

The OAI-SearchBot growth after August 2025 was not evenly distributed. Botify's vertical breakdown shows healthcare properties at +740.94%, media and publishing at +701.91%, marketplaces at +215.56%, software and internet at +204.76%, retail and e-commerce at +194.96%, and travel at +29.81%.

Healthcare and media concentration reflects the bot's function: OAI-SearchBot prioritises pages that answer concrete factual questions — clinical guidance, news events, regulatory updates, product comparisons. Travel content, which more often answers transactional queries rather than informational ones, saw the smallest proportional increase. This is fundamentally different from training crawls, which require volume across all content types regardless of question-answerability.

ChatGPT-User Is Declining

While OAI-SearchBot surged, ChatGPT-User — triggered when an AI assistant browses a URL live at a user's request — dropped 28% between December 2025 and March 2026. As OAI-SearchBot builds a broader index, the AI assistant increasingly serves answers from cached indexed content rather than fetching pages in real time during conversations. For servers, that reduces load. For site owners, it means freshness is now governed by OAI-SearchBot's re-crawl cadence rather than by on-demand requests. A product update, a price correction, or a retracted claim needs to reach the search index through the bot's next crawl, not through the next user query that hits your origin.

PerplexityBot's Coverage Gap

PerplexityBot's 0.06% unique-page coverage stands in sharp contrast to GPTBot's 3.6%. That 60:1 coverage ratio between two widely used AI citation bots means any content more than two clicks from a well-linked homepage has a low probability of appearing in Perplexity search citations. The bot's on-demand retrieval pattern — fetching pages as user queries trigger them — creates a feedback loop: pages that are frequently cited get re-fetched and refreshed; pages that are never cited because they are hard to reach stay invisible. Shallow site structure and strong internal linking matter far more for PerplexityBot coverage than they do for GPTBot or OAI-SearchBot.

What This Means for Site Owners

Differentiate your robots.txt. GPTBot and OAI-SearchBot are independently controllable. User-agent: GPTBot with a disallow blocks training crawls only; User-agent: OAI-SearchBot controls search-citation indexing independently. A wildcard User-agent: * disallow knocks out both. If you want to allow search citations but deny training data use, the mechanism exists — you need to split the rules rather than relying on a single directive.

Treat freshness as an indexing problem, not an origin problem. With ChatGPT-User requests down 28%, the AI assistant rarely fetches your pages live during user sessions. Updates reach end users through the search crawler's index. Keeping your sitemap <lastmod> values current and ensuring changed pages receive internal links from high-authority sections gives OAI-SearchBot clearer re-crawl signals than updating the origin alone.

Prioritise structured, factual content in high-OAI-SearchBot verticals. In healthcare and media — where OAI-SearchBot activity grew over 700% — you are competing directly for AI citation share. Schema markup (Article, FAQPage, MedicalWebPage), visible authorship, clear publication dates, and answers to specific questions in the first two paragraphs are documented signals that improve citation probability. Sites with heavy JavaScript rendering or content buried under accordion menus face a structural disadvantage against competitors whose pages answer questions immediately in the HTML.