Why Do the AI Bots Crawling Your Site Never Actually Send You Any Visitors?
AI bots now outnumber humans in server logs — but most of them are training crawlers that will never send a referral. Here's what the crawl-to-click data says about which bots actually matter.
Have bots finally overtaken humans on the web?
For the first time in internet history, bots outnumber people on the web. Imperva's 2025 Bad Bot Report found that automated traffic crossed 51% of all web requests in 2024 — the first time bots have outnumbered humans in a decade. By June 2026, that figure had reached 57.5% bots versus 42.5% humans. Even more striking: AI crawlers have now crossed 50% of all crawler traffic, finally surpassing traditional search bots like Googlebot in combined volume.
So who exactly is in your logs — and is any of this actually good for you?
Where does this data come from?
This post draws from Imperva's 2025 Bad Bot Report (covering trillions of bot requests globally), edge-network traffic data covering roughly 20% of all global web traffic published in late 2025 and H1 2026, and independent research from bot log analysts tracking user-agent behaviour across large publisher datasets. Where figures differ between sources, they're labelled with their period.
Which bots are actually showing up in your logs?
When you break down crawlers by share of unique HTML pages hit, the landscape in late 2025 looks less AI-dominated than the headlines suggest. Googlebot still leads at around 11.6% of unique page requests. GPTBot comes in at 3.6%, Bingbot at 2.6%, and ClaudeBot and Meta-ExternalAgent both sit at about 2.4%.
PerplexityBot — behind arguably the fastest-growing AI search product out there — barely registers at around 0.06% of unique pages. That puts GPTBot at roughly 60 times the crawl footprint of PerplexityBot by this measure. Growth rates between these platforms can sound comparable in press coverage, but their actual presence in your server logs is vastly different.
More importantly: raw crawl volume tells you almost nothing about whether these bots are useful to you.
What are AI bots actually doing when they visit?
Not all AI crawlers are doing the same job. Break down AI crawler traffic by purpose for May 2026 and the picture shifts dramatically:
Just over 51% of AI crawler requests are for model training — bots reading your content to feed into a future model. These crawlers have no mechanism to send you traffic. They serve no users, they can't click links, and they will never recommend your site. Another 9.3% is search and index building, which does feed AI recommendation systems that can eventually refer traffic — but typically on a lag of weeks or months. The remaining roughly 39% is "user-action" crawling: a real person asked an AI assistant a live question, and the assistant fetched your page in real time to answer it.
That 39% is genuinely the fastest-growing segment — user-action crawling grew over 15x in 2025 alone. But the key context is that it's growing from a very small base. Even as total AI bot traffic grew 187% year-over-year in 2025 while human traffic grew just 3.1%, most of that growth is in the training bucket that will never drive a referral.
How many crawls does it take before you see an actual click?
The most honest metric here is the crawl-to-click ratio: how many page requests does a crawler generate for every one visitor it sends back to your site?
Googlebot sits at about 4.9:1 — for every five page reads it sends you roughly one click. PerplexityBot, because Perplexity is primarily a search-and-answer product rather than a training platform, comes in at around 111:1. GPTBot sits at roughly 904:1 as of May 2026 (down from ~1,276:1 in Q1 2026 — a meaningful improvement). And ClaudeBot? As of May 2026 it was running at around 10,300:1 — ten thousand page reads per referral click. It was substantially higher earlier in 2026 before a search product launched that added citation links.
Ten thousand to one is a staggering number. But the trajectory is the important story: platforms that launch retrieval products with proper citation URLs see rapid improvement in their crawl-to-click ratio, sometimes by 80–90% within months. The gap is a function of product design, not crawl architecture.
How fast is all of this changing?
Very fast. GPTBot alone grew 305% year-over-year between May 2024 and May 2025. Overall AI bot traffic was up 187% in 2025 while the rest of the web barely budged. AI bots have expanded from less than 1% of all bot traffic in 2022 to over 22% by mid-2026. User-action crawling — the category that actually matters for visibility — grew by over 15x in a single year.
That growth is creating real infrastructure load. On some publisher sites, AI crawler traffic now exceeds Googlebot traffic in raw request volume. The question of whether any of it translates to visibility depends entirely on what category those crawlers fall into.
What should you actually do with this?
Stop optimising for crawl volume, start optimising for crawl-ability. A high rate of GPTBot visits tells you your content is readable and accessible. It doesn't predict whether any AI product will recommend you. What matters is whether the search-and-retrieval crawlers — PerplexityBot, the Bing-AI variants, user-action crawlers — can read your content cleanly. Focus on those.
Don't block all AI crawlers the same way. Blocking training-only crawlers is a defensible choice if your concern is data licensing. But blocking search-index crawlers and user-action bots cuts off exactly the pipeline that could put you in AI recommendations. Before you update robots.txt, check your actual access logs and look at which specific user-agent strings are there, not just which company they belong to. The same company often runs multiple crawlers with completely different jobs.
If your pages require JavaScript to render, none of the above matters. Most AI crawlers don't execute JavaScript. They request the raw HTML. If your page renders as a blank document without client-side JS, every crawl request is generating server load with literally zero upside — you're not getting indexed, not getting trained on, and not getting cited. Fixing static renderability, even for key landing pages, beats every robots.txt configuration decision you could make.