Which AI Bots Are Actually Crawling Your Site — and What Do They Want?

22% of all verified bot traffic hitting websites right now comes from AI crawlers. Four years ago that number was under 1%. But here's the thing: the bot doing the most damage to your server logs probably isn't the one you'd expect based on the headlines.

The findings below draw on three data sources: a 48-day server log study by WISLR covering February–March 2026 across a mid-market SaaS site (12,099 raw AI bot requests), HUMAN Security's 2026 State of AI Traffic & Cyberthreat Benchmark Report covering aggregate traffic across their network, and monthly crawler ranking data from WebSearchAPI's July 2026 report. None of these is a perfect controlled experiment, but they point in consistent directions.

Who's actually at the door?

Which bot shows up most often in practice? GPTBot tends to dominate the headlines, but when you look at actual server log data for a typical site, the answer is often different.

In the WISLR study, Meta-WebIndexer logged 1,833 requests over 48 days — nearly double the second-place bot. ChatGPT-User (923 requests) and a major AI search crawler (549 requests) were both well ahead of PerplexityBot (456) and GPTBot (187). That last number surprises most people. But it tracks: GPTBot is a bulk training crawler that hits a site hard once, then may not return for months. It's a sprinter, not a regular commuter.

The practical implication: you can't assess your AI crawler exposure just by checking which bot name you recognise from SEO coverage. Meta-WebIndexer isn't in many people's mental model of "AI crawlers," but it was the single most active bot in this dataset by a wide margin.

Are training crawlers really taking over?

There's an important functional divide in AI crawler traffic that doesn't always come through clearly in aggregate stats. Training crawlers are batch-mode bots that harvest web content to improve future AI models. Real-time fetchers are bots triggered by live user queries — they retrieve specific pages to help construct an answer right now. These two groups behave differently, deserve different access policies, and have different downstream consequences for your content.

According to HUMAN Security's 2026 report, training crawlers accounted for 41.1% of AI crawler requests in January 2026, rising to 53.3% by June. That's a significant shift inside six months. What's driving it? Partly new entrants building fresh training datasets, partly the continued scaling of existing models. Real-time inference crawlers are still growing in absolute terms — just more slowly than their training counterparts.

If you're optimising purely for real-time retrieval — hoping your pages get pulled into AI-generated answers — you're already dealing with the minority of AI crawler traffic. The majority of requests in H1 2026 were adding content to future training sets, not answering someone's question right now.

Does crawl timing actually matter?

Yes, and more than you'd expect. The WISLR study found that 81% of all ClaudeBot requests arrived in a single 3-minute burst at 04:00 UTC. That's not a distributed crawl — it's a scheduled batch job. If your infrastructure isn't sized for that kind of spike, you're looking at a predictable surge that could cause rate-limiting or increased error rates right before your European morning traffic starts.

ChatGPT-User showed the opposite pattern: requests spread throughout the day in smaller bursts, consistent with actual user-initiated retrievals. GPTBot made essentially no requests for most of the observation period, then appeared — alongside ClaudeBot — requesting sitemap.xml on the same day, suggesting a shared discovery mechanism, possibly a common third-party URL feed.

These patterns aren't just interesting trivia. If 81% of a specific crawler's traffic arrives in a three-minute window, you can plan for it, or rate-limit it intelligently without blocking it entirely. That requires log-level data, not just dashboard-level UA counts.

What's happening at the network level?

Zoom out from a single site to aggregate data and a few more figures come into focus. WebSearchAPI's July 2026 crawler rankings showed ClaudeBot at 16.28% of AI-bot traffic, with GPTBot at 9.74% — an inversion from July 2025, when GPTBot led. The competitive dynamics between major AI labs play out directly in your server logs month by month.

At the macro level, HUMAN Security found that automated internet traffic grew 23.5% year-over-year in 2025 — eight times faster than human traffic growth at 3.1%. Agentic AI traffic (bots acting on behalf of users performing tasks, not just crawling) grew approximately 7,851% year-over-year. If that number feels implausible, it's real: it started from a very small base, but the direction is unambiguous and the acceleration isn't slowing.

What should site owners actually do with this?

Treat different crawlers as different actors. GPTBot and ChatGPT-User are operated by the same AI platform but have entirely different crawl signatures, arrival patterns, and content purposes. Each major AI platform now runs at least two distinct crawler types, and some run three or more. Your robots.txt and rate-limiting rules should reflect this, not lump all AI bots into one group based on a shared company association.

Volume in logs doesn't determine impact. A real-time retrieval bot serving user queries sends far fewer requests than a training crawler, but its influence on where your content appears in AI-generated answers is disproportionately large. A bot with 187 log entries over 48 days can have more downstream visibility consequence than one with 1,833, depending on what it does with what it collects. Optimising only for bots that show up loudly in your logs is optimising for the wrong thing.

Build for burst timing patterns. Getting log-level visibility into arrival patterns — not just UA counts — is the difference between proactive management and reactive firefighting. Most page-view analytics tools strip or ignore bot traffic entirely; you need server-level or middleware-level logging with UA matching to see what's actually happening. Once you have that data, scheduling-aware rate limits become possible.

The AI crawler landscape is moving fast enough that data from six months ago needs treating with caution. Monthly monitoring isn't optional anymore — it's baseline hygiene for any site that cares about AI-driven discovery.