Which AI crawlers are actually hitting your site — and what do they want?

GPTBot grew 305% year over year. That's not a projection — it's what the traffic logs from the first half of 2026 actually show. AI bots now account for 22% of all automated web traffic, and automated traffic itself makes up 57.5% of everything hitting servers. That means roughly one in eight requests on the modern web is an AI crawler. So which ones are hitting your site, and what exactly are they looking for?

Where does this data come from?

The figures here draw on three main sources: Presenc AI's crawler volume growth study covering 2022–2026; WebSearch API's monthly AI crawler reports (April–July 2026); and US Tech Automations' robots.txt blocking analysis across the top 55,000 sites, published July 2026. Content-preference data comes from a Search Engine Journal analysis of 68 million AI crawler visits.

Is there actually a winner in the AI crawler market?

If you checked monthly, you'd get a completely different answer each time. Between April and June 2026, ClaudeBot and GPTBot flipped positions twice. April had ClaudeBot slightly ahead at 11.69% of AI-bot HTTP requests versus GPTBot's 9.84%. By May the order had reversed — GPTBot climbed to 11.48% while ClaudeBot slipped to 9.73%. Then June hit: ClaudeBot nearly doubled its share to 19.8% while GPTBot eased back to 9.4%.

The most likely explanation isn't a product launch or a search algorithm change. It's batch training runs. When a large-scale training job kicks off, the crawl traffic from that operator spikes hard for a few weeks, then subsides once the run completes. By July 2026, ClaudeBot had pulled back to 15.6% — still well above its May level, but down sharply from the June peak. For anyone trying to reason about "which AI crawler is dominant," the honest answer is that it depends on the week.

Are most of these crawlers here to train models, or to actually answer questions?

This is the more strategically important question, and the answer surprises a lot of site owners. According to network traffic data from May 2026, 51.8% of AI crawler requests are classified as training-purpose, with mixed-purpose crawls (training plus live retrieval) adding another 35.7%. Pure search and retrieval accounts for just 9.3%.

But here's what makes that framing tricky: a separate analysis of live-session and retrieval bots put agentic and retrieval crawls at 59.4% of AI crawler activity, with pure training crawls at 29.6%. The discrepancy comes down to how you classify bots that run as separate user agents for different purposes. Most major AI operators now run distinct user-agent tokens for training versus live query retrieval — but the boundary between "training" and "answering" is murkier than the names suggest.

What this means practically: a meaningful chunk of AI bot requests are real-time queries responding to actual users. If you block an AI retrieval bot, you're not just keeping your content out of a future training dataset. You're stopping an AI assistant from surfacing your content when someone asks a directly relevant question today.

What does content do AI crawlers actually prefer?

Unlike traditional search engine crawlers, which work from structured crawl queues and follow link graphs methodically, retrieval bots follow salience signals. An analysis of 68 million AI crawler visits found that sites with review integrations saw a 97.1% crawl rate versus roughly 58% for comparable sites without them. Long-form articles and product detail pages attract the most AI crawler visits; checkout flows, account-gated pages, and JavaScript-only render targets are largely skipped.

This has a direct implication for JavaScript-heavy sites. Training bots almost never execute JS. Retrieval bots sometimes do, but not reliably. If your product descriptions, pricing, or key editorial content only appear after JS hydration, a large share of AI crawler visits is returning near-blank HTML.

Are sites blocking AI crawlers — and is it actually working?

Of the top 55,000 sites analyzed in July 2026, 19.1% block at least one AI crawler via robots.txt. Among news publishers specifically, 82.4% restrict at least one crawler. GPTBot is the most-blocked single bot, appearing in 5.52% of all DISALLOW rules across the sample — blocked outright by 8,432 tracked sites.

But the blocking data hides an uncomfortable finding: roughly 95% of citations from the leading AI assistants come from sites that have blocked training crawlers. The training block reduces your presence in future model weights, potentially. It does essentially nothing to stop current AI assistants from citing pages that were indexed before the block was set, or that are being fetched by a retrieval-only bot variant you haven't addressed in your robots.txt.

This is partly a taxonomy problem. Most major AI operators now run separate bot identities — one for training data collection, one for search indexing, one for live query retrieval. Blocking the training token doesn't block the others. If you want to stay out of training data specifically while remaining accessible to AI assistants, you need to address each token separately.

What does this mean for your site?

AI-referred traffic to US retailers grew 393% year over year in Q1 2026, with AI-sourced visitors converting 42% better than visitors from other channels. The cost of being invisible to AI search is no longer theoretical — it shows up in revenue.

A few things worth acting on:

Audit your robots.txt against the full token list. A single entry blocking "GPTBot" misses the search-indexing and live-retrieval variants. If you want selective access — allow answers, block training — you need to be specific about which tokens you allow and which you don't.

Assume AI crawler traffic can double in weeks. The June 2026 ClaudeBot surge wasn't announced. If your origin has no caching layer between it and AI crawlers, a training job from a major operator can generate more requests in a month than your site normally sees in a year.

Render your key content server-side. If the HTML that lands at a crawler is largely a JS shell, you're invisible to the bots that don't execute scripts. That includes most training crawlers and a significant share of retrieval bots. This isn't a "fix it eventually" item if AI visibility is a priority for your site.

The distribution of AI crawler traffic will keep shifting. The operators running these bots are scaling fast, launching new variants, and adjusting crawl strategies based on model training cycles that aren't public. What's more stable is the underlying pattern: if your content is crawlable, structured, and server-rendered, it'll be accessible to whatever mix of bots shows up next month. If it isn't, no amount of robots.txt tuning will help.