Numbers from the AI crawler edge.
Twice-weekly research on AI crawler accessibility, bot traffic, measurement, and the MCP protocol. Every post is grounded in fresh data.
AI Crawler Failure Modes: 57% of Top Sites Invisible, Lazy Load Fails for All
A May 2026 audit of the top 1,000 most-visited sites found 57% serve content only after JavaScript runs. The AI crawlers now generating referral traffic execute none of it. Here is the failure-mode breakdown.
Analytics Platforms and AI Referral Traffic: What Each One Misses in 2026
GA4's AI Assistant channel launched May 13, 2026 with no historical backfill. Matomo 5.8 ships dedicated chatbot reports. Neither captures mobile app dark traffic. Here's what each platform sees—and what it doesn't.
41% of Production MCP Servers Have No Auth — and Only 8.5% Meet the Spec
Forty-one percent of official MCP registry servers have no authentication. Only 8.5% implement OAuth 2.1, the standard the spec mandates. Censys counted 21,000+ internet-accessible services by May 6. Here is the data and what to fix first.
AI Bot Traffic in 2026: Training Crawlers, Scrapers, and the Referral Engine That Actually Pays
New cross-dataset research reveals a striking gap: the bot sending 13,500 pages of crawl requests returns just one referral visit. Here is what the 2026 data says about who is really hitting your site.
41% of Pages Have JSON-LD — AI Crawlers Only Read the Server-Rendered Half
71% of audited sites use at least one schema type, but only 22% pass validation cleanly. AI crawlers can only read the server-rendered fraction. Here is the adoption gap and what to do about it.
AI Crawlers Hit Your Origin 4,200 Times a Day — Most of It Bypasses the CDN
GPTBot sends a median 4,200 requests per site per day. ClaudeBot sends 1,800. With 70–100% of those being unique URLs, your CDN cache barely touches training-crawler traffic — and every miss hits your origin.
AI Crawler Signal Reliability: From Raw UA Strings to Verified Attribution
5.7% of requests bearing AI crawler user-agents are fake; for ChatGPT-User the rate is 1-in-6. Meanwhile client-side analytics captures none of it. Here is the four-layer verification stack.
MCP's Action-Tool Majority: When 65% of Live Endpoints Modify State
Analysis of 177,436 deployed MCP tools shows 65% now execute actions rather than reading data — up from 27% sixteen months ago. Here is what the shift means for site owners running MCP endpoints.
Zero Out of 500 Million: How AI Crawlers Actually Handle JavaScript
Vercel and MERJ tracked 500 million GPTBot requests and found zero JavaScript executions. Here are the three rendering failure modes making sites invisible to AI crawlers.
The AI Crawler Leaderboard Changed Twice in 60 Days: Reading the Churn
GPTBot led, then ClaudeBot overtook it in April, then Bytespider nearly tripled by May. Monthly swings of this magnitude make single-bot optimization strategies obsolete before you ship them.
Three Layers of AI Traffic — And Why Your Analytics Only See One
GA4 logged 5 referrals while server logs showed 56 requests — 9% coverage. That ratio captures the structural gap between client analytics and what AI systems actually do to your site.
MCP's Open Relay Problem: 40% of Live Servers Require No Authentication
A May 2026 scan of 7,973 live MCP servers found 40.55% accept tool calls without any credentials—while write-capable tools now constitute 65% of the ecosystem. Here is what that exposure looks like in practice.
Training Crawlers vs. Browsing Agents: Two Traffic Signatures You Need to Separate
80% of AI bot traffic is training crawlers that never send referrals. Only 20% is real-time browsing agents that deliver actual users. Here's how to read the difference in your logs.
OAI-SearchBot Overtook GPTBot: The Inversion Site Owners Missed
After August 2025, OAI-SearchBot generates more server-log events than GPTBot across enterprise sites. The two bots serve opposite functions — and most robots.txt files still treat them identically.
llms.txt at 18 Months: 7.4% of Fortune 500 Have It, AI Crawlers Rarely Fetch It
After 18 months of industry conversation, only 7.4% of Fortune 500 companies have published llms.txt — and the AI crawlers it targets rarely fetch the file at all.
One-Third of Your Product Page Is Invisible to AI Crawlers — And That Traffic Converts 42% Better
Adobe's Q1 2026 retail benchmark: product detail pages averaged 66% machine readability. Meanwhile, AI-sourced traffic converts 42% better. This is the gap site owners need to close.
GA4's AI Channel Captures One-Third of AI Traffic. Here's Where the Rest Goes.
GA4's AI Assistant channel launched May 13, 2026 and captures roughly 33% of AI-referred sessions — the ones with a referrer header. The other 67% land in Direct. Meanwhile, AI crawlers require a different measurement stack entirely.
AI Traffic Quality Reversed in 12 Months. Most Analytics Stacks Still Can't See It.
AI-referred traffic swung from converting 38% worse to 42% better than all other sources in 12 months. Adobe's Q1 2026 data puts RPV 37% higher. GA4's new AI channel captures only part of the signal.
The Crawl-to-Refer Gap: What 50 Billion Daily AI Requests Actually Return
An edge network dataset covering 50 billion daily AI crawler requests shows one platform at a 38,000:1 crawl-to-refer ratio in July 2025 — down 87% from its early-year peak.
MCP Authentication in 2026: 25% Open, 53% on Long-Lived Keys
Six months after OAuth 2.1 became mandatory in the MCP spec, 25% of production servers have zero authentication and 53% rely on static API keys.
GPTBot Coverage Fell from 84% to 12%. The Bot Replacing It Can't Execute JavaScript.
GPTBot's web coverage fell from 84% to 12% as publishers blocked training crawlers. OAI-SearchBot, the replacement, now reaches 55% of the web and tripled after a major AI model launch — but it cannot execute JavaScript.
robots.txt Controls 60% of Sites Tried. The Compliance Gap Is Real.
79% of top news sites now block AI training crawlers via robots.txt. Publishers that blocked lost 23% of monthly traffic. A network security report caught one AI search provider deploying stealth crawlers to evade the blocks.
The AI Referral Attribution Gap: 70% of Sessions Are Misclassified as Direct
70.6% of AI-referred sessions arrive in GA4 as Direct traffic. Here's the data behind the gap, a platform-by-platform breakdown, and a three-layer measurement approach.
MCP's Tool Mix Has Inverted: 65% of Agent Tools Now Modify External State
A UK AI safety lab analyzed 177,436 MCP tools and found action-taking tools grew from 27% to 65% in 16 months. Meanwhile, 41% of live MCP servers have no authentication at all.
llms.txt Is Live on 2% of the Web. AI Crawlers Still Can't Execute JavaScript.
The 2025 Web Almanac found llms.txt on 2.13% of sites—39.6% auto-generated by a plugin. Analysis of 500 million AI crawler requests found zero JavaScript execution. Only one of these gaps causes active content delivery failures today.
Training Crawlers Take 80%, Return Almost Nothing: The AI Bot Split Hiding in Your Logs
In 2025, training crawlers consumed 80% of all AI bot traffic while some returned fewer than one referral per 38,000 pages crawled. Here is who is crawling, why, and what the mix means for your server.
The AI Crawler Mix Has Flipped: Bot Traffic Patterns in Mid-2026
Bots now account for 52% of all web requests — the first time automated traffic crossed the majority line. Bytespider nearly doubled its share in a single month. Here is what the 2026 AI crawler mix means for your infrastructure.
Training Crawlers vs. Browsing Agents: What the 2026 Bot Traffic Split Means for Your Site
Training bots account for 80% of AI crawler requests, but the agentic layer grew 15x in 2025. The crawl-to-refer ratio reveals which bots actually send visitors back — and the gap is 120x between top crawlers.
Training Crawlers vs. Live Agents: Anatomy of the AI Bot Layer
80% of AI crawler traffic is model training—not live user queries. But live agents grew 21x in 2025. Blocking training bots without understanding the split is the wrong tradeoff.
The Crawl-to-Referral Gap: What 50 Billion Daily AI Requests Tell You
AI crawlers now generate 50 billion requests per day, yet ClaudeBot crawls nearly 24,000 pages per referral it sends back. Full breakdown of who is crawling, why, and what the traffic mix means for your AI visibility strategy.
The Crawl-to-Refer Gap: What AI Bots Take vs. What They Send Back
One AI crawler reads nearly 24,000 of your pages for every visitor it sends back. We break down the 2026 bot-traffic data: who is crawling, why, and what site owners should actually do about it.
70.6% of AI-Referred Clicks Land as "Direct" in GA4 — Here's How to Model What You're Missing
AI assistants strip referrer headers on the majority of outbound clicks. The result hides in your Direct channel. Server logs are the leading indicator that lets you recover the signal.
llms.txt Adoption in 2025: 0% of the Top 1,000 Sites, ~10% of the Broader Web
A scan of the top 1,000 most-visited sites finds zero valid llms.txt files. A separate study of 300,000 domains puts adoption at ~10%. Here's what the data actually shows — and why the gap matters for AI crawler accessibility.
0 of the Top 1,000 Sites Use llms.txt — And AI Crawlers Still Can’t Render JavaScript
Not one of the world’s top 1,000 most-visited sites publishes an llms.txt file, and every major AI crawler skips JavaScript execution entirely. Data from June 2025 scans and a 500-million-request study explain the two-gap problem.
AI Crawlers: Traffic Almost Level With Google, Referrals Nowhere Close
AI training crawlers consumed 4.2% of all HTML requests by late 2025, approaching Googlebot's 4.5% share. Their crawl-to-referral ratios tell a different story: over 1,000 pages crawled per click sent back.