Which AI Bots Are Actually Crawling Your Site — And Are You Handling Them Correctly?

39,000 requests per minute from a single AI bot. Not a DDoS. Not a scraping attack. Just one AI assistant answering user queries about a publisher site that happened to be mentioned in a trending story — that's the peak load logged by Fastly's 2025 threat research. But here's a question worth sitting with: do you actually know what type of AI bot hit you last week? Or have you been treating them all as the same thing?

If you're like most site operators, you've probably set some robots.txt rules, maybe blocked a few user agents, and called it a day. The problem is that the AI bot ecosystem has split into three very different categories — training crawlers, search indexers, and real-time fetchers — and the rules that apply to one category don't apply to the others at all.

Where does this data come from?

The figures in this post are drawn primarily from Fastly's August 2025 AI bot threat research, which covered a 90-day window of traffic across Fastly's global CDN network. We cross-referenced those findings against a major CDN's verified bot traffic metrics published for May–June 2026, and against Botify's analysis of more than 7 billion server log files. The AI companies themselves don't publish crawl volume data — all publicly available figures come from CDN operators and large-scale log analysis.

Training crawlers: the bots that are definitely hitting your site

Training crawlers — GPTBot, ClaudeBot, Meta-ExternalAgent — are the systematic archivers. They crawl the web to build datasets for AI model training. Think of them like Googlebot, except instead of building a search index, they're building a training corpus.

These bots generate the lion's share of AI bot traffic. Fastly found that crawlers (training and indexing combined) account for roughly 79% of all AI bot HTTP traffic. Within the training crawler category, the vendor breakdown might surprise you: Meta leads with 52% of AI crawler traffic, well ahead of Google's 23% share and the OAI-affiliated training crawlers at 20%.

The crucial counterintuitive point: blocking a training crawler has essentially no effect on your AI search visibility. GPTBot feeds model training. It has no direct link to whether you appear in AI-generated answers or search citations. Blocking it is a legitimate intellectual property decision — but it won't change how often an AI assistant recommends your site to users.

Search indexers: the bots you probably haven't paid enough attention to

Search indexers are a separate, smaller category — but they're the ones that actually determine whether you appear in AI search results.

OAI-SearchBot is the bot that builds the index which AI-powered search queries against. Block OAI-SearchBot and you're not in those answers. Block GPTBot instead and your AI search presence is unaffected. Most robots.txt guides — and most site owners' instincts — get this backwards.

PerplexityBot operates on the same principle for Perplexity's search. The user-agent strings for these bots are distinct and independently configurable. You can allow OAI-SearchBot while disabling GPTBot in the same robots.txt file, or vice versa. Which one you actually want to block depends entirely on your goals — and those goals are different.

Why does this matter? Many sites that implemented AI blocks during the 2024–2025 wave of blocking adoption targeted GPTBot based on name recognition rather than function. Those sites may have inadvertently removed themselves from AI search results without realising it.

Real-time fetchers: the spike you didn't plan for

Real-time fetchers are the newest category and the one most likely to cause actual operational problems. When a user asks an AI assistant to retrieve and summarise a specific page — "what's the cancellation policy on this site?" or "what did this outlet publish about that story today?" — the assistant dispatches a real-time fetch request.

This traffic doesn't follow a schedule. It spikes with query volume and can be brutal when a domain goes viral on an AI platform. 39,000 requests per minute in a single burst, according to Fastly's logged data — against one site, during one query surge. Rate-limiting at the edge by user agent is far more effective here than robots.txt, because well-behaved training crawlers respect robots.txt but a burst of fetcher traffic needs active infrastructure-level handling.

What's changed in 2026?

The fastest shift this year has been the growth of search indexer and fetcher bot categories relative to training crawlers. Botify's analysis of over 7 billion log files found that AI-related crawl volume roughly tripled in the months following the last major model launch cycle — and that growth wasn't evenly distributed. OAI-SearchBot grew proportionally faster than the GPTBot training crawler, suggesting the ecosystem is shifting from bulk training data collection toward real-time web access.

Separately, CDN data from June 2026 shows significant month-on-month volatility in crawler market share. The bot mix isn't stable — it's shifting faster than most site operators update their robots.txt files.

So what does the blocking picture actually look like?

25% of the top 1,000 websites block GPTBot. 79% of top news and media sites block AI training bots via robots.txt. But the key question isn't who's blocking — it's what they're blocking and whether it aligns with their goals.

Most of these blocking decisions were made when GPTBot was the dominant AI bot and the training/indexing/fetcher distinction wasn't widely understood. Since then, the ecosystem has fragmented significantly. A robots.txt file written 18 months ago probably doesn't reflect what's actually showing up in your server logs today.

Three things worth doing this week

Check your server logs by user agent. Don't rely on analytics dashboards — they either lump AI bots together or exclude them entirely. Grep for GPTBot, ClaudeBot, Meta-ExternalAgent, OAI-SearchBot, PerplexityBot, and ChatGPT-User separately, and see what's actually visiting your site. The distribution may be very different from what you've assumed.

Separate your robots.txt rules by intent. If you want to appear in AI search results, explicitly allow OAI-SearchBot and PerplexityBot. These are not the same bots as the training crawlers, and a generic AI block catches them. If you want to block training but stay visible in AI search, you need separate Disallow rules for each user agent.

Add rate-limit rules for real-time fetcher user agents. Training crawlers are predictable and arrive in low-volume, scheduled patterns. Real-time fetchers are neither — and if your site gets mentioned in a trending AI query, you could absorb thousands of fetches with very little warning. Edge-level rate limiting by user agent is the right tool, not robots.txt.

The gap between how most sites think about AI bots and what's actually in their logs is wider than it's ever been. And with the ecosystem moving as fast as it is, that gap is probably getting bigger.