Numbers from the AI crawler edge.
Twice-weekly research on AI crawler accessibility, bot traffic, measurement, and the MCP protocol. Every post is grounded in fresh data.
Is your best-converting traffic hiding in your Direct bucket?
70.6% of AI-driven visits arrive in GA4 without a referrer. That misclassified traffic converts 4x better than your paid channels — here's what you're actually missing and how to start measuring it.
What Do AI Crawlers Actually See When They Visit Your Site?
42% of JavaScript-rendered content is invisible to the AI crawlers hitting your site. Here's which crawlers can read what, why llms.txt is spreading, and what actually fixes the problem.
What Do AI Crawlers Actually See on Your Site? The 2026 JavaScript Gap
Forty-two percent of JavaScript-rendered content is invisible to AI crawlers. And the llms.txt file the industry told you to create for them? Ninety-seven percent get zero crawler visits.
Is GPTBot Hitting Your Site 4,200 Times a Day? (Probably Yes)
If you're relying on Google Analytics to track AI crawler traffic, you're missing almost all of it. Here's what server logs actually show — and the numbers are bigger than most site owners expect.
Can AI Crawlers Actually Read Your Website — or Are You Just Assuming They Can?
97% of websites that published an llms.txt file got zero AI crawler requests against it. And 69% of AI crawlers can't run JavaScript at all. Here's what the access data actually shows.
If 40% of MCP Servers Can Do Something Irreversible, How Many of Those Are Intentional?
A June 2026 audit of 2,031 MCP servers found that 40% expose tools that agents can't easily undo — deleting records, executing commands, wiping indexes. The question is how many of those were deliberate design decisions.
Is Your Site Invisible to AI Crawlers? The Data Might Surprise You
97% of llms.txt files get zero AI crawler traffic, and AI crawlers can't render JavaScript either. Here's what your site actually looks like to the bots deciding whether to cite you.
Which AI Bots Are Actually Hitting Your Site — and Do Any of Them Send You Traffic?
AI-related requests now make up over a quarter of all verified bot traffic on major networks. But almost none of that actually sends referral clicks. Here's what the 2026 data says about what's crawling your site and why.
Can You Trust What Your AI Agent Just Read From an MCP Server?
5.5% of open-source MCP servers examined in a 2026 study contain tool poisoning — hidden instructions that hijack agent behaviour after a legitimate connection. Here's what the attack data shows and what to do about it.
Three Ways Your Content Disappears Before AI Crawlers Can Read It
69% of AI crawlers can't execute JavaScript — but the real problem is more subtle. Three failure modes hide content even after you've moved to SSR. Here's how to find them.
What Are AI Crawlers Actually Doing on Your Site — and Are Any of Them Worth Welcoming?
A 4,580-to-1 crawl-to-refer ratio across the major AI assistants. Three bot categories — training crawlers, real-time scrapers, agentic browsers — with very different volumes, cadences, and content preferences.
Is Your GA4 Hiding 70% of Your AI Traffic?
70.6% of AI-driven visits land in GA4 as Direct with no referrer header. Even after the May 2026 AI Assistant channel update, roughly 1 in 3 AI visits still go unattributed. Here is what the data shows.
Are You Blocking the Wrong AI Bots?
59% of AI bot hits come from bots trying to answer real user questions, not train a model. Here's what 48 days of server logs and industry data reveal about which bots are actually hitting your site — and what your robots.txt is missing.
What Does Your New GA4 AI Channel Actually Miss?
GA4 added an AI Assistant channel in May 2026. But 1 in 3 AI-assisted visits still land in Direct, Perplexity isn't captured, and crawlers are invisible to GA4 entirely. Here's what the gap looks like.
If 41% of MCP Servers Have No Auth, Is Your Agent Actually Secure?
A scan of 518 registered MCP servers found 41% accept tool calls with zero authentication. Here's what that actually means if your agent connects to third-party servers in production.
Your Site Does SSR. So Why Are 46% of Your Pages Still Invisible to AI Crawlers?
Adobe analyzed over a trillion retail site visits and found up to 46% of retailer pages aren't readable by AI. Most of those sites already use server-side rendering. So what's actually breaking?
ClaudeBot Is Now Bigger Than GPTBot. So Which One Actually Matters More for Your Site?
In July 2026, ClaudeBot led GPTBot on every single day of the month — 16.28% of AI-bot traffic to 9.74%. Does market share actually tell you which crawler matters? The answer is more complicated than the rankings suggest.
Which AI Bots Are Actually Crawling Your Site — and What Do They Want?
AI bots now account for 22% of all verified bot traffic. But the bot hitting your logs hardest probably isn't the one you'd expect. Here's what 48-day server log studies and H1 2026 aggregates reveal.
Is Your Analytics Platform Missing the Biggest Traffic Category on the Web?
AI bots now account for 34.30% of all automated web traffic — ahead of search crawlers for the first time. But your analytics platform records zero of it. Here's what's actually in your server logs.
Which AI Bots Are Actually Hitting Your Site — and Why They're Not All the Same?
AI bots now outnumber humans in web traffic. But the mix — training crawlers, search agents, real-time fetchers — matters more than the total. Here's how to read the data.
Which AI bots are actually crawling your site right now?
ClaudeBot just overtook GPTBot for second place in AI crawler traffic, leading on all 31 days in July 2026. But 47% of that traffic is for training — not sending you visitors. Here is what the data shows.
Is Your AI Bot Traffic Actually Worth Anything?
Most AI crawlers scrape your site hundreds of times for every visitor they send back. Here's the data on which bots are actually worth allowing — and which are pure server overhead.
Why your analytics are lying to you about AI search traffic
Server logs show 4–6x more AI crawler visits than client-side analytics. Here’s why referrer headers are broken for AI traffic, which signals actually work, and how to build a measurement stack that gives you the real picture.
Are AI Crawlers Taking More Than They Give? The Numbers Behind the Gap
AI crawlers now account for 22% of all verified bot traffic — but ClaudeBot returns one referral for every 1,917 pages it crawls. So what exactly are they giving back?
Why Does It Take 24,000 Crawled Pages to Get One AI Referral?
AI crawlers consumed 22% of verified bot traffic by mid-2026, yet the referrals they send back are thin. Here's what server logs reveal about the real economics of AI crawler visits.
What's Actually in Your Server Logs? The AI Bot Mix Has Changed Completely
AI bots now account for 22% of verified bot traffic — but over half are training crawlers that send almost nothing back. Here's what the numbers say about who's actually hitting your site.
Three types of AI bot are hitting your site — and only one of them follows your rules
AI crawler traffic split into three distinct categories in 2026. Training bots, search indexers, and real-time agent crawlers each behave differently — and your robots.txt was built for just one of them.
Why Are AI Bots Crawling Your Site 286,000 Times Without Sending You a Single Visitor?
AI crawlers now make up 22% of all bot traffic — and one major crawler was fetching 286,000 pages for every single referral it sent back. Here's what the real numbers look like, and what you should do about it.
Why Can a Third of the Web's Top Sites Not Be Read by AI Crawlers?
A May 2026 study of 274 fintech homepages found 17% deliver zero content to AI crawlers. 36% serve less than 80% of their page. Here's what's causing it — and why robots.txt isn't the full picture.
Is Your AI Traffic Data Actually Right? The Overcounting and Undercounting Problem in 2026
Your server logs overcount AI bots by up to 11% due to UA spoofing, while GA4 misses 70% of AI-driven visits. Here's why both gaps exist and how to close them.
Which AI Bots Are Actually Reading Your Site — And What Do They Want from It?
GPTBot hits some sites 4,200 times a day, but training crawlers now account for less than half of all AI bot traffic. Here’s who’s actually in your logs in 2026 and what each bot is after.
Are Rate Limits the Biggest MCP Problem Nobody's Talking About?
MCP rate limits are quietly becoming the top production failure mode in 2026. Three real-world case studies show how fast even conservative agent sessions hit 429s — and why some servers don't log it when they do.
Why Is 97% of Your llms.txt Being Ignored?
Most sites rushing to add llms.txt are doing it while their content is invisible to AI crawlers. Here's what the data actually shows — and what to fix first.
Why Do AI Crawlers Hit Dead Pages Four Times More Often Than Googlebot?
Analysis of billions of requests on a major hosting network found GPTBot and ClaudeBot each return 404 errors on about 35% of fetches — nearly four times Googlebot's rate. Here's what that gap reveals.
Why Are AI Crawlers Eating Your Bandwidth Without Giving Anything Back?
AI bots now make up 22% of all bot traffic — but less than 10% of those requests are the kind that can actually surface your content in an AI assistant answer. Here is what the data shows.
Is Your Analytics Actually Measuring AI Traffic — Or Just the Lucky 30%?
GA4's new AI Assistant channel launched in May 2026 — but 70.6% of AI-driven sessions still land in Direct. Here's what your analytics is missing, platform by platform.
Who's Really Crawling Your Site — and Should You Care Which Bot It Is?
AI crawlers now make up 22% of all verified bot traffic, up from roughly 10% two years ago. So which bots are actually growing, and what are they doing when they land on your site?
Is Your MCP Server Actually Reachable — or Just Running?
The July 2026 spec stripped session state from MCP, making servers cheaper to run than ever. But there’s still no standard way for an AI assistant to find yours automatically.
What Are Those Thousands of Daily AI Bot Hits Actually For?
Training crawlers now account for 44.5% of all AI crawler traffic — up from 35.7% a year ago. Here is how to read the three-way bot split and make smarter decisions about which ones to let in.
Your llms.txt Is Published. Has Any AI Crawler Actually Read It?
8.7% of the top 1,000 websites publish an llms.txt file — but 97% of those files receive zero requests from AI crawlers. Here's what's actually blocking AI visibility on most sites.
Is 70% of Your AI Traffic Invisible to GA4?
70.6% of visits referred by AI assistants arrive with no referrer header — landing silently in your "direct" bucket while converting at over 4× the rate of typical direct traffic.
Is 70% of Your AI Traffic Hiding in Plain Sight?
Most AI-sourced visits land in GA4 as Direct — 70% arrive with no referrer at all. Here's why it happens, how bad the numbers are, and what you can actually do about it.
Are AI Crawlers Actually Sending Traffic Back to Your Site?
Bots now drive 57.5% of web requests, but all AI chatbots together send just 0.29% of search referral traffic. Here's what the crawl-to-referral gap means for your site.
Why Does 70% of Your AI Traffic Show Up as Direct?
Three-quarters of clicks from AI assistants land in analytics as ‘Direct’. Here’s why your GA4 is missing most AI referrals — and what a three-layer measurement fix looks like.
Why Are 41% of MCP Servers Basically Open to Anyone?
Three independent security scans of public MCP servers in 2026 all landed on the same number: 41% have no authentication at all. Here's what's behind it, and what it means if you're exposing tools via MCP.
Is Your robots.txt Blocking the Wrong AI Bots?
Sixty percent of reputable websites now block at least one AI crawler. But research shows 71% of those blocks also hit the search bots that drive referrals — the opposite of what most site owners intend.
What's actually in your AI bot traffic? The training/retrieval split your logs are missing
Your server logs are full of AI crawler hits — but fewer than 3% of them originate from a real user asking a question. Here's how to tell the bots that matter from the background noise.
Which AI bots are actually reading your site right now?
ClaudeBot overtook GPTBot three months running through July 2026. But training bots and search bots aren't the same thing — here's what your logs are actually telling you.
1,917 Pages Crawled, One Referral Back: What's Your AI Bot Traffic Really Worth?
AI crawlers now account for nearly a third of all web traffic — and the ratio of pages crawled to visits sent back is staggering. Here's what the data shows.
Which AI bots are crawling your site right now — and do any of them actually send you traffic?
AI crawlers generated 187% more traffic in 2025 while human visits crept up just 3%. But with crawl-to-refer ratios hitting 24,000:1, the real question isn’t who’s crawling — it’s whether any of them matter.