Who's actually crawling your site right now — and what are they after?
AI bots now account for 22% of all verified bot traffic. Here's the breakdown of which crawlers are hitting your site, how fast they're growing, and what they actually want.
Something odd is happening in your access logs right now. Of every non-human request hitting your server, roughly one in five is no longer a search engine you'd recognise — it's an AI crawler. That share was under half a percent four years ago. If your monitoring hasn't registered a major shift in bot composition, your tools probably aren't looking in the right place.
So: who's actually showing up, what are they trying to read, and does it matter what they find when they get there?
How did we measure this?
The numbers in this post draw from three datasets: DataDome's February 2026 analysis of 7.9 billion AI agent requests across their network, the HUMAN Security 2026 State of AI Traffic benchmark report, and Presenc AI's longitudinal crawler volume research covering 2022–2026. Where we cite percentage figures, they come directly from those sources — no extrapolation.
Who's actually behind all these requests?
Across the January–February 2026 window DataDome analysed, three bot families account for nearly all identifiable AI crawler traffic.
The single largest group — GPTBot plus its companion agents OAI-SearchBot and ChatGPT-User — accounts for around 69% of observed AI bot requests. Meta-ExternalAgent sits second at roughly 16%. ClaudeBot and its real-time browsing sibling account for around 11% combined. PerplexityBot and a growing tail of newer entrants split the remaining 4%.
What's striking about this isn't just who's at the top — it's how stable the ranking has been even as total volumes accelerate. The relative split across these three major groups has held for over a year. That consistency is useful if you're deciding which crawler behaviour to prioritise: the answer hasn't flipped recently.
How quickly is this growing?
Faster than most monitoring setups have adapted for.
AI-driven search visits hit 15.6 billion in Q1 2025. Twelve months later that was 27.4 billion — a 42.8% year-on-year increase on an already large base. Automated traffic overall is now growing roughly eight times faster than human traffic, according to HUMAN Security's benchmark data.
The numbers for individual bots are harder to ignore once you look at them. GPTBot grew 305% year-on-year between mid-2024 and mid-2025. PerplexityBot started from a much smaller base but logged a 157,490% increase in raw requests over the same period. That second number isn't a typo — it reflects how small PerplexityBot's footprint was in 2024 and how aggressively Perplexity has been expanding its indexing operation.
What exactly are they looking for?
The crawl purpose split is shifting. In mid-2026, training-purpose crawlers — bots building datasets for model training rather than generating real-time answers — account for around 53% of AI bot traffic. That share is declining, because real-time search and agentic crawlers are growing faster and absorbing more of the volume.
That distinction matters more than it might seem. Training crawlers sweep breadth-first across your sitemap on a batch schedule, prioritising text-dense pages over anything requiring interactive rendering. Real-time indexers follow user queries more opportunistically, tending to hit the exact pages appearing in AI-generated answers — a much narrower target that changes with what users are actually asking about.
Agentic traffic is the fastest-growing category of all. These are bots that don't just index — they navigate pages, follow links, fill forms, and make decisions based on what they find. HUMAN Security measured 7,851% year-on-year growth in agentic AI traffic. In absolute terms it's still a fraction of total AI bot volume, but that trajectory is the kind of thing you want a logging strategy for before it becomes significant in your access logs.
Sector-wise, e-commerce and retail absorbs around 20% of AI bot volume, with real estate at 17% and travel and hospitality at 15% behind it. These sectors have large catalogs of structured, frequently-updated content — exactly the kind of material AI systems are built to ingest and surface to users making decisions.
What should you actually do about this?
The most common question we hear is "should I block AI crawlers?" For most sites, the more useful question is whether AI crawlers are seeing anything useful when they arrive.
Most AI crawlers fetch HTML without executing JavaScript. If your pages only render meaningful content after client-side hydration, these bots are leaving with an empty body tag. They log a visit. They return no usable content. You're invisible to any AI-generated answer that draws on indexed material. For e-commerce sites, AI-referred visitors are already converting 42% better than non-AI channels and generating 37% more revenue per visit (Q1 2026 retail data). Invisibility to AI indexers has a measurable cost — one that will only grow as AI search referrals scale.
The second problem is measurement. Most analytics stacks still don't correctly separate AI assistant real-time browsing (where a live user query triggers a web request) from batch training crawlers (which may never drive a referral visit at all). The user-agent strings are different — GPTBot versus ChatGPT-User, for instance — but most off-the-shelf tools bundle them under a generic "bot" category or miss them entirely. If you can't distinguish a category growing 305% from one growing 7,851%, you can't make sensible decisions about which to optimise for.
The third issue is how fast the mix changes. The top three operator groups have held their relative positions for a while, but PerplexityBot's trajectory shows how fast a new entrant can go from negligible to significant. A monitoring setup that only tracks the crawlers you already know about — or that relies on weekly batch exports rather than real-time edge data — will consistently lag behind what's actually happening on your infrastructure.
The practical baseline is to log user-agent strings at the edge on every request, before they reach your application layer. Not primarily as a security measure — most of these bots are well-behaved and follow robots.txt — but as an intelligence measure. If you can see who's crawling, how often, and which paths they hit, you can actually answer the question this post started with. And you can start making the changes that will matter to the AI systems increasingly deciding which sites are worth surfacing to users.