Which AI bots are actually visiting your site — and why do so few of them send traffic back?

Here's a number that tends to stop people short when you mention it in analytics conversations: roughly four in five AI bot requests hitting the average website right now are from training crawlers — bots that are systematically downloading the internet for model development, not the conversational AI assistants that might ever surface your content in a real response.

That matters more than it sounds. The bots generating the most noise in your server logs are optimising for coverage and data volume. The bots that could actually generate a referral visit are a much quieter minority. Confusing the two leads to optimisation decisions — and blocking decisions — aimed at entirely the wrong target.

So what's actually in those bot logs?

Who's doing all the crawling?

Network-level data covering roughly 6.5 trillion monthly HTTP requests (Q2 2025) shows the AI crawler market is a lot more concentrated than most analytics dashboards suggest.

Meta's training crawlers — Meta-ExternalAgent and FacebookBot — account for 52% of all AI bot traffic in that dataset. Google's AI training bot, Google-Extended, takes a further 23%. GPTBot sits at 20%. Every other AI crawler — PerplexityBot, ClaudeBot, Bytespider, and dozens of smaller players — shares the remaining 5%.

Three organisations are responsible for 95% of AI crawling volume. If you're looking at your logs and seeing what looks like a diverse mix of bot names, the request counts next to those names are almost certainly much smaller than you'd expect relative to the total.

This concentration is fairly stable across content categories. News publishers, code repositories, and documentation sites attract more aggressive crawling in absolute terms — dense, updateable, indexable content is what training pipelines want. But the top-three dominance holds regardless of category.

Are training crawlers and retrieval bots actually the same thing?

No — and this is where most site owners' mental model runs into trouble. "AI bots" covers two quite different traffic types:

Training crawlers run systematic sweeps to collect data for building and updating models. They're doing batch processing — no JavaScript execution, no dynamic content rendering, just raw HTTP fetches across as many URLs as possible. They have no interest in whether you rank in search results; they care whether your content is text-dense and reachable. Meta-ExternalAgent is the clearest current example.

Retrieval bots activate in real time when a user asks a question. ChatGPT-User, PerplexityBot, and similar agents go fetch live web pages when a conversation needs current or specific information. These are the bots that create referral opportunities — if their answer summarises your content and links back to you, there's a real chance a user clicks through.

The Q2 2025 data puts training crawlers at around 80% of all AI bot traffic, with retrieval bots at roughly 20%. But the commercial value is almost entirely in the smaller slice.

You can also spot the difference in timing patterns. Training crawlers generate flat, machine-timed request sequences with no detectable daily rhythm. Retrieval bots follow user query patterns — volume rises during business hours and spikes around breaking news events. If you're graphing AI bot traffic by hour and one user agent looks like a flat line while another looks like a human, you're seeing this distinction directly.

How many visitors do the biggest crawlers actually send back?

This is probably the single number most worth knowing: the crawl-to-referral ratio — how many pages a bot crawls for every real visitor it sends back to your site.

Network analysis from early 2025 measured these ratios directly. ClaudeBot's ratio stood at 286,000 crawls per referral in January 2025 — for every single visitor it directed somewhere, it had already crawled 286,000 pages across the web. By mid-year that improved substantially to around 38,000:1 as the underlying AI product incorporated more live-search features. GPTBot sat at roughly 1,276:1 over the same window. PerplexityBot was the outlier in the other direction: just 118:1 — it crawls selectively and sends a meaningfully higher share of users back to original sources.

The uncomfortable practical read: the bots showing up loudest in your log files are the ones returning the least traffic per crawl. The bot actually responsible for referrals is quieter, more selective, and accounts for a much smaller slice of total AI bot requests.

These ratios are also moving quickly. GPTBot's ratio has fallen substantially as the ChatGPT product added live-search capabilities — it's behaving more like a retrieval bot than a pure training crawler. A 7x improvement in ClaudeBot's ratio happened within a single six-month window. The log volumes you're seeing today aren't a reliable proxy for the referral value you're getting.

Is blocking achieving what people think it is?

The blocking rates have climbed sharply. Academic research tracking robots.txt configurations across reputable sites found the share explicitly disallowing AI crawlers went from 23% in September 2023 to nearly 60% by May 2025. Among major news publishers, 79% now block at least some AI crawlers.

GPTBot is specifically the most-blocked: it appears in around 5.52% of all active DISALLOW rules across surveyed sites, and roughly 5.6 million websites name it in their blocklists — up from 3.3 million in late 2024, a 70% increase in roughly six months.

A few things worth understanding before you add to that list:

Reputable crawlers do respect robots.txt. All four major AI crawlers — GPTBot, ClaudeBot, Perplexity's bot, and Meta-ExternalAgent — have documented compliance with disallow rules. Blocking them works, at least for the named user agents. The practical question is whether the outcome is actually what you want.

Training crawlers and retrieval bots use different user agents. If you block GPTBot but not ChatGPT-User, you've opted out of training data collection but not live-search crawling. Many sites running broad AI-bot blocking rules are catching the training crawlers but leaving the referral-generating retrieval bots untouched — sometimes by accident.

There's a structural tension in who's doing the blocking. The sites most likely to maintain aggressive AI-crawler blocks are high-value news publishers and authoritative reference sources. Those are also the sites AI assistants most want to cite when answering questions. The result is that some AI assistants end up recommending thinner, less authoritative sources because the well-sourced ones are locked out.

What should you actually change?

Three practical steps that follow directly from the data:

Segment your bot traffic before drawing conclusions. Your server logs contain the user-agent strings to do this today. A basic filter that separates training-heavy crawlers (Meta-ExternalAgent, Google-Extended, GPTBot) from retrieval-heavy ones (ChatGPT-User, PerplexityBot) gives you two useful signal groups instead of one noisy aggregate. The referral numbers that actually matter come from the second group.

Write specific blocking rules rather than broad ones. A blanket AI-crawler block in robots.txt mixes together bots that consume bandwidth without returning visitors and bots that send referrals. If your goal is limiting training data use, target training-dominant agents specifically. If you want to stay visible in AI-generated answers, keep retrieval bots explicitly allowed.

Make key content available without JavaScript rendering. Training crawlers and retrieval bots are both predominantly non-rendering HTTP clients. Content that requires JavaScript execution to appear — single-page apps without server-side rendering, lazy-loaded article bodies, dynamically injected structured data — is largely invisible to both. Content-dense pages with structured headings and proper server-rendered HTML consistently attract more AI crawler attention and are easier for retrieval bots to summarise accurately. That doesn't require a full architectural overhaul; it requires ensuring key content exists in the initial HTML response.