What's Your AI Crawler Traffic Actually Worth? It Depends on Which Bot Is Visiting

Here's a number that should give you pause: as of June 2026, automated traffic accounts for 57.5% of all HTML requests on the web. For the first time in years, bots outnumber humans. A significant chunk of that is AI crawlers. But are those crawlers actually working for you, or are they just passing through to collect what they need?

How we looked at this

The data here draws on traffic analysis from a major internet infrastructure provider monitoring billions of daily AI crawler requests across their global CDN network, per-bot request counts published by a developer hosting platform, and a published GEO research report comparing crawl-to-referral ratios across the leading AI bots. Figures cover the 12-month window from May 2024 to May 2025 unless noted otherwise.

Which bots are actually winning the AI crawler race right now?

The AI crawler pecking order has reshuffled more in the past 12 months than in the previous two years combined.

GPTBot went from 5% of AI crawler traffic in May 2024 to 30% by May 2025 — a 305% increase in raw request volume. Meta-ExternalAgent appeared from near-zero to capture 19% of the market in the same window. These two crawlers essentially absorbed the share that others shed.

Bytespider — the crawler associated with the team behind TikTok — had a rough year. It fell from 42% of AI crawler traffic down to just 7%. The collapse tracks closely with widespread robots.txt blocking campaigns across major publishers following legal pressure in late 2024. When enough sites add a Disallow rule, even a dominant crawler's share craters fast.

ClaudeBot dropped from 11.7% to 5.4% with raw request volume falling 46%, though the cause looks more like a deliberate shift in crawl focus than external pressure.

PerplexityBot grew 157,490% year-over-year. It started from essentially nothing and still holds only 0.2% of total AI crawler traffic — but a trajectory like that tells you something about where real-time AI search is heading.

Are these crawlers here to help you rank, or just to harvest what they can?

This is probably the question most site owners aren't asking often enough. The answer depends almost entirely on what kind of crawl is happening.

Over the 12 months to mid-2025, 80% of AI crawler traffic was classified as training-purpose — harvesting content to build or update AI models. Just 18% was search-indexing (the kind that surfaces your pages when someone queries an AI assistant), and only 2% was user-action crawling (live browsing triggered by someone actively asking a question right now).

The training share has been growing steadily. In July 2024 it accounted for 72% of all AI crawling. By July 2025 that figure had climbed to 79%. Search-indexing has fallen in parallel, from 26% down to under 18%.

This doesn't mean training crawls are worthless to you. Being in the training data of widely-used AI systems likely shapes how those systems talk about your brand, products, or content. But it's a slow, indirect benefit — and there's a real server cost to absorbing the traffic in the meantime.

How many of these visits actually come back as real traffic?

Here's where things get uncomfortable for anyone assuming AI crawler activity translates directly into referrals.

Googlebot maintains roughly a 5:1 crawl-to-referral ratio — about five pages crawled per human visitor returned. That reflects a decades-long model where crawling and sending traffic are directly linked. Search-oriented AI bots participate in the same loop, just less efficiently. PerplexityBot, the most referral-friendly of the major AI crawlers, shows a ratio around 193:1. GPTBot comes in at roughly 900:1.

Training-focused crawlers sit in a completely different range. A published GEO analysis found ClaudeBot's crawl-to-referral ratio at 10,300:1 — that's over ten thousand pages indexed for every single visitor sent back to publishers.

The implication is fairly direct: if a training-mode crawler is hitting your site heavily, you're absorbing bandwidth costs and having your content ingested at scale, with near-zero direct return in the form of visitors. The only payoff is indirect — being cited by an AI assistant at some unknown future point.

What does this actually mean for managing your crawler traffic?

A few things become clearer once you look at AI crawler activity by purpose rather than treating it as a single undifferentiated category.

Not all AI crawlers are equivalent. A search-indexing bot with a 193:1 crawl-to-click ratio and a training bot with a 10,000:1 ratio will both show up in your logs as "AI crawler traffic," but they represent fundamentally different relationships with your content. Configuring crawler rules at the purpose level — not just the user-agent level — is where the real leverage is.

User-action bots are where the most measurable value sits. These bots fire when a real user is querying an AI assistant in real time. User-action crawling grew 15x in 2025 — the fastest-growing category in the data — and it's the segment most likely to actually send visitors back to you. Serving these bots fast, structured, semantically rich HTML is the version of AI optimisation that has a direct feedback loop you can measure.

High-volume training crawlers deserve a deliberate decision. If a training-mode bot is generating thousands of requests per day on your server with a near-zero referral return, that's not a theoretical concern — it's a real cost question. Rate-limiting or blocking a pure training crawler won't affect your search performance in AI assistants. It just means your content isn't feeding a pipeline that isn't giving anything back.

The 57.5% figure is dramatic. But the more useful question is what percentage of your AI crawler traffic is actually working toward something that returns value to you. For most content-heavy sites, the honest answer is considerably smaller than the raw numbers might suggest.