Why are you still scoring AI search like it’s a keyword ranking?

Ask one of the big AI assistants to name the best tools in your category. Now ask it 99 more times. The odds that any two of those answers list the same set of brands? Under 1 in 100.

That's not a glitch you can prompt your way out of. It's how these systems work. And it means the single "AI visibility rank" your shiny new dashboard reports is a snapshot of one dice roll. Most brands are measuring AI search with instruments they inherited from the keyword-ranking era, and those instruments are quietly lying to them. If that's you, you're behind the curve, and the rest of this post is about what catching up looks like.

A quick word on where the numbers come from. The headline consistency figure is from a 2026 study by SparkToro co-founder Rand Fishkin and Patrick O'Donnell, CTO of the AI-tracking startup Gumshoe.ai. The variance figures come from a July 2026 arXiv paper and a working paper on SSRN. The market-size numbers come from SE Ranking, SimilarWeb and Conductor. Several of these are vendor reports, and I've flagged the ones with a reason to spin.

Does an AI "ranking" mean anything if you only looked once?

Fishkin and O'Donnell ran 2,961 prompt tests. Six hundred volunteers each ran 12 brand-recommendation prompts, between 60 and 100 times, across three of the leading assistants. The result: less than a 1-in-100 chance that two responses to the same prompt return the same list of brands. The membership changed, the order changed, even the number of brands returned changed.

Compare that with a vendor test from MaxAEO, which found only 34% of prompts produced the same shortlist across 12 runs. The two don't agree, and you shouldn't read them as the same measurement: SparkToro counted exact-list matches over up to 100 runs, MaxAEO counted looser shortlist stability over 12. Different method, different number. But they point the same way, and that's the point. Fishkin's takeaway was blunt: stop tracking "position" and start tracking appearance rate, meaning out of 100 runs, how often your brand shows up at all. So if you checked your "rank" on Tuesday, checked again Friday, and it moved, you probably learned nothing about your content and everything about the random seed.

If the answer is mostly noise, what is your dashboard actually measuring?

Here's the finding that should unsettle you. A July 2026 arXiv study decomposed the variation across 12,933 AI brand answers into its sources. Run-to-run resampling explained about 34.8% of the variance. The exact phrasing of the question explained roughly 26.5%. The brand's own identity, the one thing you actually control, explained about 1.5%.

Read that again. The single biggest lever on whether you appear isn't your brand or your content. It's whether the user happened to phrase the question slightly differently. A working paper on SSRN by Jan Ehrlinspiel, Malte Landwehr and Tomek Rudzki makes the operational version of the same argument, and it's right there in the title: prompt tracking works, but not as a one-prompt measurement system. Build your whole visibility score on one "canonical" prompt and you've overfit the number to that exact wording. Most dashboards do exactly that, then charge you monthly to watch it wobble.

Why do two "AI visibility" tools score the same brand differently?

Because they're dividing by different things. An AuthorityTech analysis found that 54% of AI answers name no brand at all.

So when a tool computes your "share of voice," it matters enormously whether the denominator is every answer or only the answers that named some brand. One tool divides by all runs; another divides by the brand-naming runs. Your score can double or halve without a single word of your content changing. That's not a rounding difference, it's a different metric wearing the same label.

This is where the named skeptics get sharp. Seer Interactive has called a standalone AI visibility score a vanity metric, the same trap the search industry fell into with keyword rankings a decade ago. The measurement body AMEC, in its guidance on generative engine optimisation, warns that counting citations without tying them to business outcomes just rebuilds AVE-style vanity metrics, the discredited "advertising value equivalent" numbers that PR spent years walking back. The lesson both land on is the same: if your vendor can't tell you its run count and its denominator, you don't have a metric. You have a mood ring with a subscription fee.

But AI search is still a rounding error, so why spend on measuring it?

This is the honest objection, so let's take it seriously. By SimilarWeb's numbers, AI assistants drive roughly 0.24% of all web traffic, and organic search still sends vastly more, with one comparison putting it near 345 times as much. If AI is a fraction of a percent of your visitors, why instrument it at all?

Three reasons. First, trajectory. SE Ranking's 2026 study found AI-driven traffic grew about 16x between 2024 and 2026. A channel sitting at a fraction of a percent and growing like that is precisely the one you want a clean baseline for before it hits 5%, not after. Second, the size estimates don't even agree with each other. Conductor's benchmark puts AI at 1.08% of traffic, more than four times SimilarWeb's figure. When credible firms disagree by 4x on how big a thing is, "too small to measure" is doing a lot of unearned work. The honest move is to measure it better, not wave it off. Third, value and visibility aren't the same as clicks. Semrush pegs an AI-search visitor at roughly 4.4x the value of an average organic one, and a separate analysis found a brand recommended by ChatGPT was about 2.5x more likely to earn a visit. More importantly, being named in an AI answer is mostly zero-click. You can shape a buyer's shortlist and never get a referral your analytics will ever see. Measure only the click and you miss most of the influence. "Too small to matter" quietly swaps today's traffic for today's influence, and both are moving fast.

So what should you actually do on Monday?

Kill the single-number rank. Replace "what position is my brand?" with appearance rate: run each priority prompt 50 to 100 times and report the share of runs that name you, with a confidence interval, not a lone point.

Test the question, not just the answer. Track a spread of paraphrases for every prompt that matters, because the wording moves the result far more than your brand does. The arXiv split was 26.5% versus 1.5%; your tracking should reflect that.

Interrogate your vendor's denominator. Ask any visibility tool two questions: how many runs per prompt, and what is the denominator for share of voice. If they can't answer both, their score isn't comparable to anything, including itself last month.

Separate "named" from "clicked." Report appearance in answers, which is zero-click influence, apart from referral traffic in your analytics. They measure different stages of the same funnel, and conflating them is how the last decade's dashboards got AI so wrong.

Baseline now. A rough, well-documented appearance-rate baseline this quarter beats a precise-looking rank that's really just tracking the random seed.

None of this is exotic. It's just measurement that matches how the thing being measured actually behaves. The brands that build that baseline this year will know whether their GEO work is doing anything. The ones still reading a single AI rank off a dashboard will be guessing, confidently, for another year.