Why more prompts beat more repeats in AI visibility tracking

If you want a more precise AI visibility number, add prompts before you add repeats. Running the same prompt again and again feels like a bigger sample, but the answers are so alike that most of the extra runs tell you almost nothing new. The maths for this is old, standard and rarely applied to AI visibility tracking.

This matters because the choice is usually framed the other way. Tools and plans are compared on how many answers they collect, and a set run 100 times sounds far more robust than one run once. It is not, and the gap is large enough to change what you can honestly report.

Why do repeats of the same prompt add so little?

Because whether your brand appears is mostly a property of the prompt, not of the run. Ask "best CRM for a 12-person agency in the UK" ten times and the answers will vary in wording, but the set of brands an assistant considers for that question stays much the same. Ask ten different questions and you sample ten different slices of how buyers search.

Statisticians call answers like this clustered. Each prompt is a cluster, and answers inside a cluster are correlated. The strength of that correlation is the intra-cluster correlation, or ICC: 0 means every run is independent, 1 means every run is identical.

The cost of clustering is measured by the design effect, from survey sampling:

Design effect = 1 + (runs per prompt − 1) × ICC

Divide the number of answers by the design effect and you get the effective sample size: how many independent observations those answers are worth.

What does that do to the numbers?

Take a set of 100 prompts, a true mention rate of 25%, and an ICC of 0.7, a mid-range assumption for brand mentions. The margin of error is the usual 95% interval for a proportion, computed on the effective sample size.

Read the third and fourth rows together. The same 10,000 answers look like a ±0.8-point measurement if you ignore clustering, and are really a ±7.1-point one. Ninety-nine in every hundred repeats added almost nothing.

Now compare the second row with the last. Both collect 500 answers. Spent as five runs of 100 prompts, they give ±7.4 points. Spent as 500 different prompts, they give ±3.8 points: nearly twice as precise for the same number of calls.

The ICC matters, but not enough to rescue repeats. At a lower correlation of 0.4, five runs of 100 prompts are worth about 192 observations and ±6.1 points, still well behind 500 distinct prompts.

Your daily tracker is already repeating prompts

Most AI visibility trackers run each prompt once a day on each assistant. Over a month, that is 30 answers per prompt, and the monthly figure on the dashboard is an average over all of them.

Those 30 answers are repeats. With 50 prompts, a month produces 1,500 answers, but at an ICC of 0.7 the design effect is 1 + 29 × 0.7 = 21.3, and the effective sample is about 70 observations. The margin of error on the monthly mention rate is roughly ±10 points, not the ±2 points that 1,500 independent answers would suggest.

Daily answers do drift over time, as models and their sources change, so the true correlation across days may be lower than 0.7. That is a reason to report the margin you can defend, not to assume the smaller one.

Is there any point in running a prompt twice?

Yes, for a different reason: ranking stability. A single run can put two brands in the wrong order by chance. In a 2025 study of repeated LLM evaluations, Alvarado Gonzalez and colleagues found that a second run removed about 83% of the rank inversions a single run produced, while barely reducing the standard error. Repeats buy a steadier ordering, not a more precise rate.

Research on brand answers specifically points the same way. Dmitrij Żatuchin decomposed the variance in 12,933 AI answers about 20 brands across 8 languages and 3 models. A repeat past the fifth reduced the relevant error by only 0.0003, and adding languages and models reduced it far more than adding repeats. His conclusion: reliability is bought by spreading across languages and models, not by repeating one prompt.

Why not just set temperature to zero?

Because it does not make answers repeatable. Atil and colleagues ran five models configured to be deterministic across eight tasks and found accuracy varying by up to 15% between identical runs. Thinking Machines Lab sampled one prompt 1,000 times at temperature zero and got 80 different completions, identical for the first 102 tokens and then diverging.

The cause is in how models are served. Providers batch requests together, batch size changes with load, and common GPU kernels give slightly different results for different batch sizes. You cannot switch that off from the prompt, which is why AI visibility is better described, as Schulte, Bleeker and Kaufmann put it, as a distribution rather than a single-point outcome.

What to do

1. Count effective observations, not answers. Divide your answers by the design effect. If you do not know your ICC, assume 0.7 and show the range from 0.4 to 0.85.
2. Spend budget on prompts first. A set of 50 or more distinct, buyer-shaped prompts beats a set of 20 run many times. See how the Prompt Fit Score treats set size.
3. Run each prompt at least twice before comparing brands or topics, for a stable ordering.
4. Spread across assistants and markets rather than repeating one prompt on one assistant.
5. Report a margin of error with every visibility number. A 5-point month-on-month change on a ±10-point measurement is not a change.
6. Reset the baseline when a model changes. A model update can move every prompt at once, which no amount of repetition inside the old model will show.

If you are not sure how big your set is in effective terms, the free RateMyPrompts scorecard flags sets too small to detect a change and shows how the prompts spread across topics. The companion piece on building a prompt set from buyer language covers what those extra prompts should be.