Which AI bots are actually crawling your site in 2026 — and are any of them sending traffic back?

AI crawler traffic hitting the average website grew 757% in 2024. But the raw volume number doesn't tell you which bots are actually showing up, whether they're likely to bring anyone back with them, or which ones half the internet's biggest sites have quietly blocked. Here's what 12 months of data actually shows.

Where this data comes from

The numbers below draw on three main sources: network-level bot traffic analysis from a major CDN provider covering 2024–2026 (spanning tens of millions of sites), Imperva's 2025 Bad Bot Report (their 12th annual study, published April 2025), and a robot-crawl study across approximately 140 million websites conducted by a major SEO analytics platform. Where specific time windows differ between sources, they're called out inline.

So who's actually showing up — and who's not?

The most overlooked story in AI bot traffic isn't the growth. It's how quickly the mix changes. A crawler that dominated your log file a year ago might be a footnote now.

Take Bytespider. ByteDance's crawler held 14.1% of verified AI bot traffic in July 2024. By July 2025 that share had collapsed to 2.4% — an 83% decline. The reason almost certainly isn't that ByteDance stopped crawling. It's that Bytespider landed on DISALLOW lists across enormous numbers of sites in response to concerns about data use, and the crawler wasn't bypassing those restrictions.

At the other end of the scale, Meta-ExternalAgent grew from 0.9% to 7.5% of verified bot traffic over the same period — a 733% increase. GPTBot roughly doubled (4.7% to 11.7%). ClaudeBot grew from 6% to about 10%.

PerplexityBot is the most extreme case: its raw request volume grew 157,490% year-over-year. It's still a small share of total traffic (around 0.5% as of mid-2026), but no other major crawler is on anything close to that growth trajectory.

What does this mean in practice? If you built your crawler access policy twelve months ago and haven't revisited it, you may be blocking crawlers that have collapsed in volume while freely serving ones that have grown enormously. The mix shifts faster than most policy review cycles.

Are they here to train — or are they actually trying to find answers?

Not all AI bots are crawling your site for the same reason, and this distinction matters enormously for referral outcomes.

Training crawlers fetch content to build or update a model's knowledge base offline. They're not going to send you a visitor from that request. They want text, structure, and clean HTML — and then they're done with you until the next crawl cycle. Retrieval and search crawlers fetch content to answer a specific query in real time, when a user is actually waiting for a response. These have a plausible path to generating referral traffic back to your site.

In May 2026, 51.8% of AI crawler requests were classified as pure training-data collection. Add in "mixed purpose" (crawls that serve both training and retrieval simultaneously), and 87.5% of AI crawler hits primarily serve model development. Only 9.3% are pure search-intent retrieval — the category with a direct path to a referral click.

That ratio has been shifting. Back in mid-2024, training accounted for roughly 72% of AI crawler activity. The retrieval and search fraction is growing, driven primarily by real-time browsing features in AI assistants. But training still dominates — which means the majority of what you see in your bot logs is traffic that will never send you a visitor.

One CDN provider described the situation bluntly: the web is being "mined more than it's being visited." Their analysis found that certain training-focused crawlers return roughly one referral click for every 24,000 pages they read. For comparison, Googlebot's ratio is around one referral per five pages read. PerplexityBot, which is search-focused, sits somewhere around 111 pages per referral click — better than training-only bots by orders of magnitude, but still nowhere near the traditional search engine baseline.

Why is blocking accelerating so fast?

GPTBot is now the most-blocked AI crawler on the open web, appearing in DISALLOW rules on 5.89% of all websites — roughly 5.6 million domains by mid-2025, up from near-zero when the bot launched in August 2023. Among the top 1,000 websites, 35.7% now block it outright.

ClaudeBot's block rate grew 32.67% year-over-year — the fastest acceleration of any named AI crawler in the study period.

The math driving these decisions isn't hard to follow. If a training crawler is reading your content at a 24,000:1 crawl-to-click ratio with no referral return, blocking it doesn't cost you visible traffic. It reduces server load and reduces the risk of proprietary content feeding a training dataset. That calculation gets easier to make as referral data becomes more available and the ratio stays as lopsided as it currently is.

News publishers arrived at this conclusion early: 79% of top news sites now block at least one AI training crawler. The blocking has also affected content availability in AI-generated answers — but that's a separate negotiation, and one many publishers have apparently decided they're willing to enter from the position of withholding access rather than granting it by default.

What should you actually do with this?

Don't assume the bot mix is stable. The crawlers generating the most volume in your logs this month may look very different from six months ago. A quarterly audit of bot user-agent strings against current published UA identifiers is now a real operational task, not a set-and-forget configuration.

Optimise differently for retrieval bots versus training bots. PerplexityBot and real-time AI assistant browsers respond to semantic structure, explicit question-answer formatting, and clean entity definitions. Training crawlers mostly care about parseable, high-quality text. If you want to improve your visibility in AI search results — which is where referral traffic actually comes from — retrieval bots are the audience that matters, and they want something slightly different from what training bots need.

The referral numbers are growing, but from a very low base. AI-attributed referral traffic is increasing quarter-over-quarter across most verticals. But even after significant growth, it's still a fraction of organic search volume for most sites. Anyone claiming AI search traffic has already arrived at scale is probably describing a specific product vertical in a specific geography, not the general web. The trajectory is right. The timeline is longer than the announcements suggest.