Which AI bots are actually crawling your site — and why does the answer keep changing?

Fifty billion AI crawler requests hit the web every day. That's the rate running through one large network operator's infrastructure as of mid-2026 — and that's just one provider's window into what's happening. What the headline number doesn't tell you is that less than 3% of those requests are triggered by a real person actively using an AI assistant.

Data from network-level analysis covering tens of millions of sites puts live user fetches at just 2.6% of all AI crawler traffic. The remaining 97.4% is a mix of training data collection, mixed-purpose scraping, and search indexing — none of which necessarily translates into a human visiting your site.

So the real question is: which bots are doing the crawling, and are any of them actually worth caring about?

Where this data comes from

The numbers here are drawn from two sources: 28-day traffic analysis from a major web infrastructure provider covering May and June 2026, and a monthly AI crawler tracking report that independently validates bot behaviour using IP verification and user-agent fingerprinting. Both sources distinguish between training crawlers, search crawlers, and real-time retrieval agents — a distinction that matters enormously for how you interpret the data.

Who's actually showing up in your logs?

The market share picture has been surprisingly volatile. In May 2026, GPTBot held 11.48% of AI bot HTTP requests, with ClaudeBot close behind at 9.73%. By June, the story had completely changed: ClaudeBot surged 66% in a single month to reach 20% of AI crawler traffic, vaulting to the second position overall. Meanwhile Bytespider — which had seen its own surge in April — reversed hard, dropping from 10.1% down to 7.3%.

That kind of swing in a single month is worth sitting with for a moment. Googlebot's share has moved in decimal points across years. AI crawlers are moving in double-digit percentages month to month. Each operator is making independent decisions about crawl intensity based on their own training schedules and product priorities — and none of them announce it in advance.

If you've been making decisions about robots.txt based on last quarter's data, you're probably working from a snapshot that no longer reflects reality. The rankings at the top of the AI crawler leaderboard look genuinely different every 30 days.

Are training bots and real-time bots the same thing?

No — and the difference is what most analytics setups miss. A training crawler harvesting your content for a dataset and one fetching your page because a user just asked an AI assistant a question are completely different interactions, yet they often look identical in your logs without careful user-agent breakdowns.

The network-level data from June 2026 breaks it down this way: 52.3% of AI crawler requests are explicitly for model training. Another 35.7% are classified as mixed-purpose — training plus some retrieval. Search indexing accounts for 9.3%. And live user fetches, where an actual person triggered the crawl in real time during an AI assistant session, account for just 2.6%.

Practically speaking: if you see 10,000 AI crawler requests in your logs this week, roughly 260 of them might have been triggered by a real person using an AI assistant with web access. The other 9,740 were likely collection runs for training datasets or search indexing queues.

This also explains why blocking one type of crawler doesn't necessarily affect your AI search visibility. GPTBot and ChatGPT-User are separate user-agent strings with separate purposes — the training crawler and the real-time browsing agent from the same team. Disallowing one in robots.txt does nothing to the other. That distinction is still missing from most guides on the topic.

What content are they actually targeting?

Content preferences vary significantly by crawler type, and the patterns are starting to become clear from log analysis.

For bots that power AI search answers — the ones building citations for user queries — the path mix skews toward homepages, /pricing/ pages, and comparison pages (URLs with /vs/ patterns or product comparisons). These are the pages that surface in AI assistant answers to "what tool should I use for X?" style questions. GPTBot revisits these high-traffic pages on an average cadence of 2.4 days. For long-form content and blog archives, the revisit cadence is considerably slower.

Bytespider runs a distinctly different pattern on e-commerce sites: it prioritises product pages over all other content, with revisit intervals as short as 1.8 days on retail sites. This matches what you'd expect from a training-focused crawler optimised for structured product data.

At the industry level, e-commerce absorbed 26.3% of all verified bot crawl traffic in June 2026 — more than any other sector. Publishing saw a 300% jump in AI bot activity across 2025, driven by the text density of editorial content. Commerce sites saw more than 25 billion AI bot requests in a two-month window in mid-2025 alone.

What does this mean for your site?

Your highest-value pages for AI visibility probably aren't your blog. If you want your content to surface in AI assistant answers, pricing pages, comparison landing pages, and anything structured around specific use cases matter more than content archives — at least for the citation-focused bots. Homepage, /pricing/, and /vs/ comparisons are the pages getting crawled most aggressively for citation purposes.

Most of that AI crawler traffic isn't sending you visitors, and won't. The 52% that's pure training and the 36% that's mixed-purpose are extracting your content, not directing humans to it. That's not inherently bad — being well-represented in training data has genuine long-term value — but it means that AI crawler request volume in your logs is a poor proxy for AI-driven referral traffic. Don't mistake high crawl volume for high AI visibility with actual users.

The mix is volatile enough to warrant active monitoring. The gap between the top two AI crawlers flipped entirely in one month. If you're making decisions about which bots to allow or block, or which pages to optimise for AI visibility, those decisions need revisiting on at least a monthly cadence. An annual review of your robots.txt in 2026 is almost certainly already outdated.

The question "which AI bots are hitting my site?" turns out to have an answer that keeps changing. The follow-up question — "what do they actually want when they get there?" — varies by bot type in ways that matter for how you think about content strategy and visibility. Both questions are worth asking regularly, not once.