Can AI Crawlers Actually Read Your Site? The Data Is More Brutal Than You Think

Only 8.7% of the top 1,000 websites publish an llms.txt file, and here's the part that's even more sobering: of those that do, 97% receive zero AI crawler requests against that file. Zero. Meanwhile, 69% of AI crawlers can't execute JavaScript at all — which means your React-rendered homepage, your lazy-loaded product grid, and your client-side navigation are returning an empty shell to the bots that are increasingly deciding which sites to recommend. So the question worth asking is: is your site actually accessible to AI crawlers, or are you just assuming it is?

How Did We Actually Look at This?

Adoption figures come from Rankability's June 2026 crawl of the top 1,000 websites and an SE Ranking study covering 300,000 domains. JavaScript rendering capability data comes from documented analysis of major AI crawler user-agent behaviour, cross-referenced with CDN log data from Technologychecker.io's 2026 AI bot traffic reference. Where sources disagreed, we used the more conservative figure.

So How Many Sites Actually Publish llms.txt?

The llms.txt format launched in September 2024 as a structured way to tell AI systems what your site contains — think of it as a robots.txt for the retrieval era. Nearly two years in, adoption among the top 1,000 sites sits at 8.7%. Broaden the lens to 300,000 domains and it creeps up to 10.13%.

The breakdown by sector is what makes this worth paying attention to. Tech companies are well ahead — roughly 36% adoption, according to the SE Ranking data. Fortune 500 companies? 7.4%. So the sites most likely to be asked about by AI users — retail, finance, news — are the ones least likely to have any structured AI-readable signal in place.

But Does llms.txt Actually Get Read?

Here's the uncomfortable follow-on: 97% of llms.txt files get zero AI crawler requests. Not a trickle — zero. There are a few reasons for this. The standard hasn't been officially adopted by any major AI assistant's crawler. Most crawlers aren't configured to check /.well-known/ paths by default. And a lot of the llms.txt files that exist are thin — a few marketing sentences that don't contain anything a retrieval system would actually use.

The files that do get reads tend to be structured tool manifests on developer-facing sites where the AI assistant already knows to look for machine-readable endpoints. For everyone else, llms.txt right now is closer to a bet on the future than a working channel.

Can AI Crawlers Even Read Your Content Without JavaScript?

This is the more immediate problem, and it's one most teams haven't properly diagnosed. Roughly 69% of AI crawlers cannot execute JavaScript. That's not a minority edge case — it's the default behaviour for every major AI crawler except a handful of specialised web search agents.

GPTBot, the crawler behind the dominant AI assistant, doesn't render JavaScript. The same goes for the main crawlers from Perplexity, Meta, and most other AI platforms. If your content loads via a React component, prices come from an API call, or your navigation relies on client-side hydration — those crawlers see a blank page. Not a degraded experience. A blank page.

The scale of this matters. AI bot traffic now accounts for 4.2% of all HTML page requests, and GPTBot traffic grew 305% year-over-year. These are not numbers you can safely ignore and plan to address later.

What Does This Actually Mean for Your Pages?

The pages most exposed to this problem aren't necessarily your homepage — they're your long-form content, your product documentation, your FAQs. These are the pages AI assistants pull from when someone asks a question your business should be answering. And they're exactly the pages most likely to be JavaScript-rendered, because they were built with a framework that made that the default.

A quick diagnostic worth running: fetch your key pages with curl and look at the raw HTML. If you see an empty <div id="root"></div> where your content should be, your site is invisible to the majority of AI crawlers hitting it today. And if your traffic logs show AI crawler user-agents making requests, those requests are real — what those crawlers see actually matters for whether your content gets surfaced by AI-driven recommendations.

What Should Site Owners Actually Do?

The highest-leverage move right now isn't writing a better llms.txt — it's making sure your content is readable without JavaScript. For most teams that means one of three things: switching to server-side rendering for key pages, serving a pre-rendered HTML version to bot traffic specifically, or using a pre-rendering service that detects AI crawler user-agents and returns cached rendered HTML. The bot-detection approach is the lightest lift for existing frameworks — you don't need to rebuild your stack, you need to identify bot traffic and handle it differently.

If you're going to invest in llms.txt anyway (it's not wasted effort, it's a long-term bet), make it substantive. A file that describes your site's purpose in two sentences gets the same zero reads as an empty file. A file that lists your content sections, describes the types of questions your site answers, and includes structured machine-readable data has at least some chance of being useful when crawler support improves.

The underlying question isn't whether AI crawler accessibility matters — at 4.2% of HTML traffic and growing, it clearly does. The real question is whether you're addressing the right layer of the problem. Most sites aren't. They're spending time on llms.txt files that nobody reads while serving blank pages to crawlers that represent a fast-growing and increasingly consequential share of their traffic.