Why Are AI Crawlers Eating Your Bandwidth Without Giving Anything Back?

You might have noticed something odd in your server logs lately. There are bots showing up with names like GPTBot and ClaudeBot, crawling deep into your site architecture — and not the "hits your homepage once" kind of crawling. We're talking repeat visits, deep pagination, API endpoint probing, the works. The question most site owners are now asking: is any of this actually going to pay off in the form of traffic?

Short answer: it depends which bots you're looking at, and the split is starker than most people realise.

So what's the breakdown between AI training bots and search bots?

AI bots now account for around 22% of all verified bot traffic as of mid-2026, up from under 1% in 2022. GPTBot alone grew roughly 305% year-over-year between 2024 and 2025. By any measure, AI crawler volume is exploding.

But here's the thing that changes the story: not all AI crawling is the same kind of crawling. There are bots that exist to harvest your content for model training, and bots that exist to retrieve your content when a real user asks a question. The first type gives you nothing. The second type gives you a citation — which might turn into a visit.

In May 2026, only 9.3% of AI crawler requests were classified as search or retrieval-only — the kind that can actually put your URL in front of someone. 51.8% were training-only. The rest were mixed-purpose, meaning the crawler may serve both functions but doesn't necessarily give you anything measurable in return.

So when you look at that 22% figure and think "AI is paying attention to me," the more useful question is: which 22%? Because most of it isn't indexing you for discovery — it's extracting you for training data.

How lopsided is the exchange, really?

The crawl-to-refer ratio measures how many of your pages an AI system fetches for every referral visit it sends back. A ratio of 5:1 — roughly what traditional search averages — means you're giving five page-views of bandwidth for every visitor returned. That feels fair.

What's the ratio for AI crawlers?

Perplexity is the clear outlier on the reciprocal end: their crawlers average around 186 pages fetched per referral visit. Still 37x worse than traditional search, but at least you're getting something meaningful back. The heavier training-focused operators sit at roughly 848:1 and 4,580:1. That top number means you're serving nearly 4,600 page requests per single visitor sent back.

And this isn't hypothetical — infrastructure costs are real. If you run a mid-size content site serving 50,000 AI crawler requests a day at a 4,500:1 ratio, you'd expect around 11 referral visits from that operator. Eleven visits. For 50,000 page serves.

The training-heavy crawlers are categorically different from search engines. They're not building an index that points back to you. They're building a model.

Which bots are site owners actually deciding to block?

GPTBot is the most-blocked AI crawler in robots.txt, appearing in 5.52% of DISALLOW rules in Q1 2026. CCBot (5.08%), ClaudeBot (4.88%), Google-Extended (4.44%), and Bytespider (4.23%) follow. Notice what's missing from that list: PerplexityBot. The crawler that actually sends traffic back is the one site owners are least likely to block.

That's not a coincidence. Site owners who've looked at the data are making rational calls: block the extractors, allow the ones that participate.

The more surprising stat is that 84.2% of sites currently have no AI crawler policy at all. No ALLOW, no DISALLOW, nothing. The door is wide open for every bot on the list. For many smaller sites, that's fine — the crawl volumes are low enough that bandwidth doesn't matter, and the citation upside is worth having. But for high-traffic sites, the maths starts to shift. News publishers are already there: 79% of them now block at least one AI training bot. They've run the numbers.

What should you actually do about all this?

The decision framework is simpler than most guides make it. There are two things worth separating cleanly: training crawlers and retrieval crawlers.

Training crawlers fetch your content to improve a model. You get no attribution, no traffic, no citation. Whether you allow them is a question about your content licensing stance and your bandwidth budget. There's no SEO benefit to allowing them — they're not building a web index.

Retrieval crawlers are the ones tied to live AI search and answer features. These are the bots that, when blocked, make you invisible in AI-powered search results. Blocking them is a genuine SEO decision with real traffic implications.

Most major AI assistant operators now publish separate user-agent strings for training versus retrieval. If you're writing robots.txt rules, this is the split to act on. Allow the retrieval-mode agents — OAI-SearchBot, PerplexityBot, and the equivalent search-purpose variants published by other AI assistant operators — and you're opting into AI search citations. Block or rate-limit the training crawlers — GPTBot, ClaudeBot, CCBot — and you're reducing your data donation footprint without affecting citation potential.

If you can serve different responses to different bot types, or measure what each type is actually fetching, that's where the real leverage is. Understanding your crawl-to-refer ratio by bot tells you exactly which systems are worth optimising for. Right now, for most sites, Perplexity is the obvious answer. Everything else requires a harder conversation about whether the bandwidth cost is worth the speculative training upside.

The 22% figure is real. What it represents, bot by bot, is where the actual decision lives.