Why Are 97% of llms.txt Files Getting Zero AI Crawler Traffic?

Why would you spend time crafting a file specifically designed to help AI crawlers understand your site — and then have 97% of them never read it? That is the finding from Ahrefs' June 2026 analysis of server log data across 137,000 domains. Fewer than 1,100 sites received a single request to their llms.txt file in the whole of May. A file built for AI, largely ignored by AI.

The short version: AI visibility is a genuine problem, but llms.txt is not where the solution lives. Two separate issues are getting conflated here — whether crawlers can access your content at all, and whether they understand it once fetched. They need very different fixes.

Where does this data come from?

The adoption figures are from Rankability's monthly scan of the top 1,000 websites (June 2026), HTTP Archive's BigQuery dataset, and a SE Ranking study of 300,000 domains. Request-level traffic data comes from the Ahrefs Web Analytics dataset, which covers 137,000 monitored domains with server-log visibility into May 2026. JavaScript rendering behaviour is drawn from Vercel's analysis of GPTBot traffic across their network.

What does llms.txt adoption actually look like?

The headline numbers are a bit misleading. Rankability put llms.txt adoption at 8.7% of the top 1,000 websites in June 2026 — and Shopify sits at 78.1%. But that Shopify figure is not organic adoption. In the first week of May 2026, Shopify silently deployed /agents.md, /llms.txt, /llms-full.txt, and an agentic sitemap to every storefront by default, with no official press release. Tens of thousands of stores got llms.txt overnight without anyone consciously choosing to add it.

Strip that out and the organic baseline in HTTP Archive's top-10k dataset sits at 5.6%. SE Ranking's study of 300,000 domains puts the broader rate at 10.1%, but that figure catches a lot of Shopify stores too. The actual count of sites that deliberately chose to publish llms.txt is considerably smaller than the headlines suggest.

Who is actually sending requests to these files?

Of the 3% of llms.txt files that received any traffic in May 2026, who sent it? Not the AI assistants site owners are trying to reach. SEO audit tools accounted for 21.7% of requests — more than double the share of all named AI crawlers combined. Unknown bots added 14.9%. General-purpose crawlers contributed 13.1%.

As for the actual AI crawlers: GPTBot sent 4.5% of requests to llms.txt files. ClaudeBot sent 0.8%. DeepseekBot sent 0.02%. The dominant AI systems that site owners want to appear in are barely touching the file. They find content the same way they always have — through robots.txt, sitemaps, and hyperlinks — and when they happen to encounter llms.txt, it rarely gets prioritized over the rest of the crawl queue.

This does not mean llms.txt is worthless. A well-structured one tells AI assistants how to weight different parts of your site and what it is for. But it is a context document, not a discovery gateway. Publishing one will not cause AI crawlers to index content they could not previously reach.

What are AI crawlers actually seeing on your site?

If llms.txt is not the main visibility lever, what is? The crawler behavior data points to a more fundamental gap: JavaScript rendering.

None of the major AI training crawlers execute JavaScript. Vercel analyzed over 500 million GPTBot requests across its network and found zero evidence of JavaScript execution. GPTBot downloads JavaScript files in about 11.5% of its requests — but treats them as static text and never runs them. ClaudeBot shows a similar pattern, fetching JS files in roughly 24% of requests without executing them.

What does this mean in practice? For any site built on a client-side rendering framework, an AI crawler sees exactly what a browser shows before JavaScript fires: typically an HTML shell with a <div id="root"></div> and very little else. No product copy, no pricing, no structured content — just a skeleton.

A significant share of modern commercial sites depend on client-side rendering for their primary content. The shift toward single-page application frameworks over the past several years has been enormous, and most of those frameworks put minimal content in the initial server response by default. Developers building for users have no reason to prioritize pre-JavaScript HTML — users never see it. AI crawlers see nothing else.

For those sites, llms.txt is optimizing the description of a room the crawlers cannot actually enter.

What should you actually do about it?

The fastest diagnostic costs nothing. Fetch your own homepage the way an AI crawler would:

curl -A "GPTBot/1.0" https://yoursite.com/

Look at what comes back before JavaScript runs. If your key content — product names, descriptions, category structure, pricing — does not appear in that initial HTML response, it is invisible to every major AI crawler regardless of how well your llms.txt is written.

For sites on client-side rendering frameworks, the fix is server-side rendering or static generation of the initial HTML. This is not a new concept for SEO — it is the same requirement Googlebot has enforced for years. AI crawlers just have no second-pass rendering phase, so there is no partial credit for partially-rendered content.

Once the HTML is accessible, llms.txt genuinely earns its place. A well-structured file provides useful context about site structure and content authority — it works as a signal layered on top of content that crawlers can already read, not as a substitute for accessible content.

The sites most exposed right now are ones that built excellent, JS-heavy frontends for good user experience reasons, assumed their existing SEO infrastructure would carry over to AI discovery, and never checked whether it actually did.