Model updates move your AI visibility more than you do

When your AI visibility jumps or falls sharply and nothing in your marketing changed, suspect the model first. Assistants change the models behind them, often without notice, and a single update can move every prompt in your set on the same day. That is not a change in how buyers see you. It is a change in the instrument you are measuring with.

This piece shows how large those moves are, how to recognise one in your data, and how to keep it out of the trend line you report.

How much can one model update move the numbers?

Enough to swamp anything a campaign does in a month. Chen, Zaharia and Zou asked the March 2023 and June 2023 versions of GPT-4 the same questions. On identifying prime numbers, accuracy fell from 84% to 51%: a 33-point drop, on identical questions, three months apart. The same update made GPT-4 better at multi-step questions, so the change was not simply "worse". It was different.

Their conclusion is the one that matters for tracking: the behaviour of what looks like the same service can change substantially in a short time. A tracker that sends the same prompts to "ChatGPT" every day is not measuring one fixed thing. It is measuring whichever model answered that day.

Do updates move every prompt the same way?

No, and that is what makes them hard to spot. Ma, Yang and Kästner tested 10 pairs of model versions in the GPT-3.5 family on the same tasks with several prompts each. Across all prompt and model combinations, 58.8% lost accuracy after an update. More telling: 55% of the updates did not move the prompts consistently. The same update helped some prompts and hurt others doing the same job.

Their example: moving from gpt-3.5-turbo-0301 to gpt-3.5-turbo-0613 cut one prompt's accuracy by 9.6 points and raised another's by 5.1 points on the same toxicity-detection data. For a visibility programme, that means an update can reshuffle your set internally. Some topics rise, others fall, and the blended mention rate may barely move while every topic chart changes.

Is the same model at least consistent with itself?

Not fully, which sets the floor underneath all of this. Atil and colleagues ran five models ten times each on eight tasks, using settings designed to make the output deterministic. Accuracy still varied by up to 15% between runs, and the gap between the best and worst possible result reached 70%. None of the models gave repeatable accuracy across every task, let alone identical text.

So there are two layers of movement. Run-to-run variation happens every day, on the same model. Model updates arrive now and then and shift the whole distribution. Your own marketing is a third force, usually the smallest of the three in any single month.

How big a move should you believe?

Compare the size of what can happen without you with the smallest change your set can detect.

These studies measure accuracy on fixed test sets, not brand mentions. But the structure is the same as a visibility tracker: a rate across a fixed list of questions, answered by a model you do not control. If a version change can move accuracy on identical questions by 33 points, a 6-point swing in your visibility after an assistant release is not evidence of anything you did.

How do you recognise a model update in your data?

By its shape. Real changes and model changes leave different fingerprints.

• Breadth. An update moves many unrelated prompts on the same day. A real change concentrates on the topics you worked on.
• Direction. Updates often push prompts both ways at once, as Ma, Yang and Kästner found. A campaign rarely makes your visibility fall on topics you did not touch.
• Competitors. If rivals jumped or fell on the same day, the instrument moved, not the market.
• Answer shape. A new model often changes how it answers: longer or shorter replies, lists instead of prose, more or fewer cited sources. If the format changed, the model probably did.
• Timing. A step change that lines up with an assistant's release note, and with nothing you shipped, is the model.

None of these is proof on its own. Two or three together are enough to hold the number back from a report until you know more.

Why keep a stable core of prompts?

Because it is the only way to compare like with like across a break. A stable core is a set of prompts you never edit, run on every assistant, every period. When a model changes, the core shows how far the instrument moved on identical questions. The rest of the set can then be read against that shift, instead of every change being blamed on marketing or credited to it.

It also stops a quieter error. If you rewrite prompts in the same month an assistant changes its model, you have two changes and one number, and no way to separate them. Change your prompts or live through an update, but try not to do both in one reporting period.

What to do

1. Record when each run happened and, where your tracker shows it, which model answered. Without that, updates are invisible.
2. Keep a change log of assistant releases next to your own launches, so a step change can be checked against both.
3. Check breadth and competitors before reporting a sudden move. A whole-set shift on one day is a model question first.
4. Treat a confirmed update as a break in the series. Mark it on the chart, stop comparing across it, and start a new baseline.
5. Report changes only when they beat your smallest detectable change. The sample-size and cadence calculator works it out for your set, and why more prompts beat more repeats explains how to shrink it.
6. Keep a stable core and avoid rewriting prompts in the same period as a known update.

How the Prompt Fit Score treats set size, coverage and noise is in the methodology. Before the next model change lands, the free RateMyPrompts scorecard shows whether your set is large and varied enough to tell a real change from a new model.