What Does Your Site Actually Look Like to an AI Crawler?

69% of AI crawlers cannot execute JavaScript. Not "struggling with", not "partially supporting" — 69%, confirmed by testing 23 major AI agents against live sites in 2026. If your website relies on a JavaScript framework to render content client-side, then the majority of AI systems that might cite or reference your work are fetching an HTML shell and walking away. Your product descriptions, article text, and pricing tables never entered the picture.

What makes this more interesting — and more uncomfortable — is the second question: even if you've added the guidance file most people reach for (llms.txt) to help AI systems understand your site, is anything actually reading it?

Where the data comes from

The 69% figure comes from technical testing of 23 AI agents published in 2026. The per-crawler breakdown — GPTBot fetching JavaScript files in 11.5% of requests without executing any of them, ClaudeBot fetching JS in 23.84% of requests with the same result — comes from an analysis of over 500 million GPTBot requests and separate crawler log studies. The llms.txt numbers come from Rankability's June 2026 survey of the top 1,000 domains and a broader study covering 137,000 sites.

Which crawlers actually render JavaScript?

Googlebot, and almost no one else.

Googlebot uses a headless Chrome-based rendering engine that attempts to fully execute JavaScript on the pages it crawls. A client-side rendered React or Vue application looks broadly similar to what a human sees. Every major AI training and indexing crawler works the opposite way: send an HTTP request, read the raw HTML response, and stop. No second pass, no JavaScript engine firing up, no waiting for the page to hydrate.

GPTBot, ClaudeBot, and PerplexityBot all follow this pattern. GPTBot downloads JavaScript files in about 11.5% of its requests — not to execute them, but because those files appear in <script> tags in the HTML it already fetched. An analysis covering over 500 million GPTBot requests found zero evidence of JavaScript execution across the entire dataset. ClaudeBot downloads JavaScript files in approximately 23.84% of requests with the same outcome: download only, never run. One major AI crawler's official documentation confirmed this explicitly in May 2026, noting that their web fetch tool "currently does not support websites dynamically rendered via JavaScript."

This is a deliberate engineering decision, not a gap about to be patched. Running a full browser rendering engine at crawl scale requires significantly more compute than parsing HTML. AI training crawlers and real-time indexers are making explicit architectural tradeoffs when they skip it.

Is llms.txt making the difference people think it is?

The adoption numbers look impressive. From 4,088 sites in June 2025 to 36,120 by May 2026, llms.txt deployment grew 8.8× in twelve months. As of June 2026, 8.7% of the top 1,000 websites publish one.

The breakdown by sector tells a more nuanced story. Developer and tech SaaS companies adopt at 23.1% — roughly four times the general web average — which tracks with their engineering-heavy marketing teams and AI-forward audiences. Shopify merchants appear at 78.1%, but that number comes with a major caveat: Shopify pushed llms.txt to every store by default in early 2026. That's platform-driven deployment, not individual teams making a deliberate choice.

Here's the number that complicates the narrative: 97% of published llms.txt files receive zero AI crawler reads. A study of 137,000 sites found no correlation between publishing a llms.txt and being cited more frequently by AI search engines. The sites that do get AI reads — the 3% — tend to be large enterprise properties or platforms where AI crawlers already had an established relationship before the file existed.

The file is inert for most sites right now. That may shift as AI systems more consistently implement the spec, but it's where things stand.

What does a crawler actually encounter?

If JavaScript rendering is the structural problem, what does a crawler actually see when it lands on a client-side rendered page?

Three failure modes dominate:

The blank body. A pure single-page application sends an initial HTTP response that typically looks something like <html><body><div id="root"></div></body></html>. No product names, no article text, no headings. Maybe a title tag and a script reference. The crawler gets the mounting point and walks away with effectively nothing indexable.

Partial hydration. Many sites use a server-rendered shell with client-loaded content. Navigation, global header, and site title might be in the initial HTML — but the article body, product details, or pricing table loads client-side. The crawler indexes a page with a coherent title and hollow content.

Scroll-triggered lazy loading. Even fully server-rendered pages often defer below-the-fold content using Intersection Observer or similar patterns. Crawlers do not scroll. If your secondary content blocks, comparison tables, or product carousels only appear when a user scrolls to them, those elements don't exist as far as an AI crawler is concerned — regardless of whether the server rendered the rest of the page correctly.

What does this mean if you run a site?

The diagnostic is simple. Run curl -A "GPTBot/1.0" https://yourdomain.com/your-important-page and read the raw output. If your actual content is there — product descriptions, article paragraphs, whatever the page is designed to communicate — you're visible to AI crawlers. If you see a mounting div and not much else, you have a gap that affects most AI indexers.

The fix is server-side rendering or static generation for the pages that matter. If you're already on a framework that supports SSR — most modern ones do — you may be closer to a solution than you think. The priority pages are the ones you'd most want AI systems to understand: product pages, key landing pages, documentation.

On llms.txt: if your server-rendered HTML is already clean and information-rich, publishing one costs almost nothing and there's a reasonable case that AI systems will read it more consistently as the spec matures. Treat it as a second-order step. The server-rendered HTML foundation comes first.

The simplest test: does the page make sense with JavaScript disabled? If yes, AI crawlers can read it. If no, they can't — and no metadata file changes that.