If AI Bots Are Crawling Your Site, Why Aren't They Sending You Traffic?

In a 55-day study of 24.4 million bot requests across 69 websites — January to March 2026 — the ChatGPT browsing agent made 3.6 times more requests than Googlebot. If you've been optimising for AI search because you assumed that AI bots were crawling your site like a smarter Googlebot, that headline probably sounds like great news. It isn't quite, and understanding why is the whole point of this post.

Where did this data come from?

This post draws on three datasets. First: a request-log study by Alli AI covering 24.4 million requests across 69 sites between January and March 2026, verified against published bot IP ranges at the proxy layer. Second: a network-level crawler purpose breakdown for May 2026, covering tens of millions of bot requests from a major edge infrastructure provider. Third: robots.txt blocking and compliance data from Q1 2026, compiled from hundreds of thousands of website configuration files. No Wrenda-internal data was available in sufficient volume to be conclusive on its own, so these external sources form the backbone of the analysis.

Is AI bot traffic actually AI search traffic?

Here is the question that the 3.6x headline doesn't answer: what were all those requests actually for?

When you break down AI crawler activity by purpose rather than raw volume, the picture looks very different. In May 2026, network-level analysis showed that roughly 52% of AI bot requests were categorised as model training — bots collecting content to improve or fine-tune language models. Another 36% were mixed-purpose (training plus live retrieval combined). Just 9% were pure search indexing. Real-time user actions — the category most likely to result in a citation or a referral back to your site — accounted for the remaining 3%.

Put simply: around 88% of AI bot visits to your site are doing something that won't directly send you traffic today. Training crawlers are valuable to the companies building AI products, and they might shape how a model understands your brand over a longer horizon. But there is no mechanism by which a training crawl results in a visitor arriving on your page this week.

This distinction matters because a lot of the "optimise for AI visibility" conversation treats all AI crawler traffic as though it were AI search traffic. If your server logs show a spike in GPTBot visits, you might conclude you are getting good exposure in AI-powered search. In reality, you are probably getting harvested for training data.

Which bots are actually growing?

GPTBot — the training crawler — went from 4.7% of verified bot HTTP requests in July 2024 to 11.7% in July 2025. That's a meaningful rise in share, and the absolute volume increase across the same period was even sharper. User-driven AI browsing agents (the bots that fetch content on behalf of a live user query) grew roughly 15x in volume over the course of 2025 according to one major infrastructure provider's year-end review — but they started from a much smaller base, so they remain a smaller fraction of total AI bot traffic by count.

The crawlers that can actually send referral traffic are retrieval agents: they fetch your page at query time to answer a specific live question. ChatGPT-User (the real-time browsing agent, distinct from GPTBot the trainer) is the clearest example. PerplexityBot operates similarly. These bots have a fundamentally different crawl-to-referral profile because every request is tied to a live user query that might result in your page being cited. Training crawlers don't work that way — a visit today might influence a model's training weights months from now in ways that are essentially invisible to your analytics.

So why are sites still blocking the wrong bots?

The most common response to all this AI bot traffic has been to block it via robots.txt — and GPTBot has become the most-blocked crawler on the web.

In early 2023, around 5% of the top 1,000 websites blocked GPTBot. By mid-2026, that figure is approximately 25%. For news publishers it's even higher: roughly 79% now block at least one AI training bot. Looking at which bots appear most frequently in DISALLOW rules in Q1 2026, GPTBot leads at 5.52% of all rules, followed by CCBot (5.08%), ClaudeBot (4.88%), Google-Extended (4.44%), and Bytespider (4.23%).

There are two issues with this picture. First, compliance: studies have found that up to 72% of AI crawlers violate robots.txt, with one tracked cohort averaging around 156 non-compliant requests per site over a three-week window in 2025. The robots.txt mechanism was designed for good-faith crawlers. Not all AI bots operate in good faith.

Second, and more importantly for anyone focused on AI search visibility: blocking a training crawler doesn't necessarily affect how you appear in AI search results. If a model has already been trained on your content, that data doesn't disappear because you added a DISALLOW line. And blocking a retrieval bot actively hurts your AI search visibility, because retrieval-based AI systems need to fetch your page at query time to include it in an answer.

The irony is that many site owners who want more AI search exposure are blocking the very bots that would deliver it, while doing nothing to stop the training bots that are consuming their content anyway.

What does this mean for your strategy?

The first thing worth doing is splitting out your AI bot traffic by type. Training crawlers and retrieval crawlers have distinct user-agent strings and IP ranges. Most CDN dashboards and server log setups can filter on these strings. Understanding your actual composition — how much of your AI traffic is retrieval versus training — is the prerequisite for any strategy that makes sense.

Second, if your goal is AI search visibility rather than blocking AI data collection, the bots you should be thinking about are the retrieval agents. These typically fetch pages at query time as static HTTP requests — no JavaScript execution, no browser rendering. If your site relies on client-side rendering to surface important content, those retrieval bots are seeing a blank page, just like Googlebot did before you added a sitemap and a server-side render. The fix is the same: make sure your content is available in the raw HTML response.

Third, the robots.txt decision is a legitimate policy choice, particularly for publishers concerned about training data use. But be clear-eyed about what it does and doesn't achieve. Given compliance rates, it is not a reliable technical control. And applied to retrieval bots, it actively cuts into your AI search reach.

The split between training and retrieval crawlers is likely to keep widening. More AI products are being built around live retrieval rather than static training snapshots. Over time that should mean more traffic-delivering bot visits. But right now, in mid-2026, the majority of what is in your server logs is training traffic that isn't going to send you a visitor. Building strategy around the volume number alone is going to lead you in the wrong direction.