Which AI bots are actually hitting your site — and should you care about any of them?

Here's a number that might change how you think about your server logs: as of June 2026, automated traffic crossed 57.5% of all HTML web requests. For the first time, bots outnumber humans on the web. AI crawlers account for roughly 20% of verified bot traffic, with AI search bots adding another 6.5% on top. So about one in four verified bot requests right now has some kind of AI system behind it — and most of those aren't doing what you'd hope.

How was this measured?

This analysis draws on publicly reported data from a major CDN provider's network (which proxies a substantial share of global web traffic), cross-referenced with reports from traffic analytics firms covering mid-2025 through mid-2026. Where sources gave conflicting figures for the same month, the lower bound was used. No figures in this post are invented — if a number is here, there's a source for it.

Who's actually knocking?

Which bots showed up most in May 2026 data? The mix is more surprising than most SEO guides suggest.

Googlebot still leads the pack at 27.3% of AI-adjacent bot requests — a reminder that legacy search and AI crawlers now get lumped into the same category. GPTBot sits at 11.5%. ByteSpider, operated by the company behind TikTok, has climbed to 10.3%, nearly doubling its share over six months to become a significant presence most site owners have never optimized for. ClaudeBot comes in at 9.7%, with a separate dedicated search crawler from the same company adding another 2.2%.

Is this ranking stable? Emphatically not. GPTBot and ClaudeBot swapped the second and third position three times in a single quarter. Month-to-month swings of 2-3 percentage points are routine. Anyone presenting a definitive "AI crawler leaderboard" is probably using data that's already six weeks out of date.

What are these bots actually trying to do?

Here's where it gets genuinely important for your strategy — not all AI crawler traffic is the same thing, and most of it isn't what you might assume.

In May 2026, 51.8% of AI crawler requests were classified as training-only. These bots are building datasets, not answering live user questions. There's no mechanism for a training crawl to send a visitor to your site — the content goes into a model, full stop. Another 35.7% were mixed-purpose, meaning training plus some retrieval functionality. Only 9.3% were classified as search-only: bots actively grounding answers to real user queries, the category most likely to eventually refer traffic.

So if you're looking at your logs and seeing a surge in AI bot traffic and wondering why it doesn't translate to visitors in your analytics, this is the structural reason: the majority of AI crawl volume exists to improve models, not to surface your content to users right now.

Does any of this traffic come back as visitors?

This is the question most site owners are actually asking, and the data is pretty sobering.

In July 2025, one major AI assistant crawler sent roughly 38,000 crawl requests for every single user who clicked through to a source. That ratio has since improved dramatically — dropping to around 11,736:1 by March 2026, a 74% improvement in eight months. But even at that improved rate, it's staggeringly high compared to purpose-built search crawlers, where the ratio sits closer to 194:1.

The underlying reason: crawlers that exist primarily to train models don't need to attribute sources. The content is consumed to make the model smarter, and the model may later answer questions without linking anywhere. Crawlers that exist to power live retrieval — answering a user's query with cited sources right now — have a different incentive structure.

AI-referred traffic to US retailers did grow 393% year-over-year in Q1 2026, according to Adobe data. That's real growth from a real base. But it's concentrated in conversational search bots, not training crawlers. And 393% YoY on a tiny base is still a tiny number — AI search referrals are a fast-growing fraction of a fraction.

Are some kinds of sites crawled harder than others?

Are you in a sector that AI crawlers prioritize? It depends entirely on what you publish.

Shopping and general-merchandise content absorbed 31.7% of all AI crawler requests in the first half of 2026 — a higher concentration than any other content category. News and media came in second at 9.6%. AI crawlers concentrate on shopping even harder than traditional bots do: product catalogs are exactly the kind of structured, frequently-refreshed factual content that training pipelines and retrieval systems both value. If you run an e-commerce site, the bots are spending more time there than your analytics dashboard suggests.

The flipside: if your site publishes static evergreen content that rarely changes, you're probably getting lighter AI crawler attention — and what you do get is more likely to be training crawls than search-grounding crawls.

What should you actually do with this information?

A few things fall out of this data that are worth acting on.

Know your crawler mix. Most analytics tools don't break out training crawlers from search crawlers — they just report "bot traffic." Your server logs contain user-agent strings that tell you which bots are responsible. GPTBot and ClaudeBot with a training purpose are not the same as PerplexityBot in terms of their likelihood to return visitors. If you're making blocking or optimization decisions, it's worth knowing which category is actually hitting your site.

Structured data matters most for shopping content. If AI crawlers are spending nearly a third of their time in product content, that's where clean HTML, well-formed schema markup, and reliably parseable product data have the clearest payoff. A crawler that can't extract price, availability, and product name from your catalog won't use it — which means it won't appear in AI-grounded answers when someone asks "what's the best option under £80?"

Blocking is a blunter instrument than it looks. A major CDN provider reported that its customers collectively blocked 416 billion AI bot scraping requests in five months after making AI-crawler blocking the default. That's a significant number of blocked requests — but lumping all AI crawlers into a single disallow rule also blocks the search-grounding bots that have the highest crawl-to-referral ratio. The more precise move is distinguishing between bot purposes before deciding what gets through. That distinction is now available in the user-agent strings themselves, if you know what to look for.

The AI crawler landscape will keep shifting. The market shares above will look different in six months — ByteSpider's growth alone makes that near-certain. But the structural pattern — training volume dwarfing search volume, crawl rate far outrunning referral rate — has held across enough monthly snapshots to treat it as the baseline, not an anomaly to wait out.