Why your AI crawler rankings look completely different every month

57.5% of all HTML web traffic is now automated. That number — from network edge data published June 3, 2026 — flipped the human-to-bot ratio for the first time on record. But what's buried inside that bot figure is more interesting than the headline: the AI crawler that topped the rankings in April had dropped to second place by May, then the May leader fell back to half its share by June. So which bot is actually hitting your site right now — and does it even matter which one it is?

Where does the data come from?

This post draws on three independent server log studies published between March and July 2026. The first is a 48-day analysis across multiple production sites tracking hit rates for 19 distinct AI user-agents. The second is a 30-day study of agentic crawler behaviour measuring daily request volumes segmented by site authority tier. The third is a monthly AI crawler market share report aggregating traffic data across thousands of sites. All three use verified IP ranges to separate legitimate bots from spoofed requests — an important step, since spoofed crawl traffic routinely inflates raw numbers by 15–20%.

Which AI crawler is "winning" right now?

The monthly swings are the story here. In April 2026, ClaudeBot marginally led the field at 11.69% of AI bot traffic to GPTBot's 9.84%. May reversed it: GPTBot climbed to 11.48% and ClaudeBot dropped to 9.73%. Then June happened: ClaudeBot nearly doubled to 19.8% while GPTBot eased back to 9.4% — a 2.1× gap that appeared in a single month with no obvious vendor announcement attached.

The prevailing theory from crawl researchers: a large batch training run rather than a sustained architectural shift. In other words, one bot ran a big job in June and the numbers swung. If you were optimising for "the leading AI crawler" based on May's data, you'd have been caught flat-footed by June's reversal. The ranking is volatile enough that it's essentially noise for any decision with a time horizon longer than a few weeks.

How aggressively are they hitting your infrastructure?

Market share percentages say nothing about server load. For that, you need hit-rate data.

GPTBot hits high-authority sites at an average of 4,200 requests per day — 2.3× more than ClaudeBot's 1,800, and nearly 8× more than Google-Extended's 540. PerplexityBot sits between them at 980 requests per day. These are averages for sites above a certain authority threshold; smaller or newer properties see much lower numbers, with GPTBot more likely visiting weekly rather than daily for sites below roughly 30 domain authority.

That 4,200 daily request figure matters for caching strategy. If your server is generating a fresh render on each GPTBot request — common on JS-heavy frameworks — you're paying significant compute costs to serve a bot whose crawl in 80% of cases has no connection to surfacing your content to end users. A static HTML cache for known training crawlers costs almost nothing to implement and removes that load entirely.

Are those crawl requests actually helping you get cited?

This is the question most site owners don't think to ask — and the answer is surprising.

Over the past 12 months, 80% of all AI crawler traffic was for training data collection. Only 18% fed live search indexes, and just 2% corresponded to real-time user queries. That means the bulk of what appears in your server logs is training data harvesting, with near-zero predictable connection to whether your content gets cited in an AI assistant's response.

The 18% that feeds search indexes behaves differently. Those requests are smaller in volume but higher in signal: they care about canonical URLs, schema markup, clean HTML, and whether your content renders without JavaScript. Optimising for this slice is what actually moves the needle on AI search visibility.

What content types are AI crawlers actually spending time on?

Citation data from AI search results shows a clear preference for listicle-style content (33.8%) and product pages (28.0%), together accounting for nearly 62% of all cited content. Homepages appear in 11.3% of citations — a higher rate than most SEO teams budget for. The remaining share goes to blog posts, documentation, and long-form guides, but only when they're structured with clear headers and short, citable paragraphs. Dense prose with no scannable structure gets indexed but rarely cited.

Worth flagging: different AI search platforms skew differently. One major AI search platform favours product pages heavily (41.6% of its citations), while another shows a balanced split between listicles and product pages. So if you're trying to understand which content drove an AI referral, the referrer source matters as much as the content format.

So what should you actually do with all this?

Stop tracking crawlers as brands, start tracking them as categories. Monthly market share data is too volatile to build strategy on. What stays stable is the category: training crawlers (high volume, zero user impact), search indexers (lower volume, direct citation impact), and live user-request agents (smallest volume, highest-quality signal). Your optimisation priorities should map to categories, not individual bot names.

Fix the cache layer for training crawlers first. If GPTBot is hitting your site thousands of times per day and each request triggers a server-side render, the compute cost is real even if the user-visible benefit is zero. Returning cached static HTML to known training user-agents removes a meaningful chunk of overhead. It costs nothing in terms of crawler compliance.

Focus your content effort on the 18%. The slice of AI crawler traffic that feeds search results responds to exactly what good technical SEO has always rewarded: semantic HTML, structured data markup, clean URL hierarchies, and content that renders completely without client-side JavaScript. If your site serves an empty div to a bot without JS execution, you're invisible to AI search — regardless of how polished the human-facing experience is.

The market share chart will look different next month. That's fine. What you're actually building — structured, renderable, semantically clear content — doesn't need to change based on which bot happened to run a bigger batch crawl this week.