To compare two prompts fairly, change only the prompt. Keep the test inputs, the scoring rubric, the number of repetitions, the tool access, and the model settings identical for both variants. When any of those conditions drifts, the comparison stops measuring prompt quality and starts measuring the drift.
This guide is for practitioners who need a comparison they can defend — whether you are choosing between two instruction styles for a classification task, deciding which prompt to ship in a WordPress automation, or tuning a local model workflow. The procedure below is drawn from two published lessons on fair prompt comparison and is written so you can reproduce it with a spreadsheet, a script, or a manual run log.
The one-variable rule
Comparing two prompts is the same discipline as repeating a single prompt, extended by exactly one step. When you repeat one prompt, you freeze the input, the metric, and the run settings to find out whether that prompt behaves consistently. When you compare two prompts, the prompt becomes the deliberate variable — that is the entire point of an A/B comparison — but every other condition has to stay just as frozen. CodeSignal’s lesson on fair prompt comparisons states the rule directly: in a fair comparison, the prompt is the only deliberate variable.
When a condition other than the prompt is allowed to change, that condition is a confound. A confounded comparison can produce a confident-looking winner while telling you nothing about the prompts. The failure mode is not exotic: one prompt is tested on easier cases, graded with a looser metric, or given more attempts than the other, and any of those differences alone could produce the entire score gap.
What “the same test inputs” actually covers
Identical input text is the starting point, not the whole requirement. The Prompt Engineering Institute’s lesson on comparing prompts and models fairly lists the conditions to preserve when running variants against the same representative cases:
- Input text — the same cases, in the same form, for both prompts.
- Allowed sources — if either prompt may consult retrieved documents, both variants see the same source set.
- Expected behaviour — you have written down what a correct output looks like before you run anything.
- Scoring rubric — one rubric, applied to both prompts, not a stricter reading of one prompt’s output.
- Number of repetitions — equal attempts for both variants.
- Tool access — the same tools, with the same permissions, available to both.
- Time-sensitive data — held steady where possible, so one variant does not see fresher or staler context.
Two of these deserve emphasis because they are easy to violate by accident. The rubric is part of the test, not a formality: if you grade Prompt B’s outputs with more generosity than Prompt A’s, you have measured your grading, not the prompts. Repetition count is the other one. If one prompt gets five tries and the other gets one, the comparison is no longer like-for-like.
Running the comparison, step by step
- Write the comparison question first. A usable question names the variable and the outcome, for example: Does prompt B improve citation support over prompt A on the current model? Each question defines which variable may change and which conditions must remain fixed.
- Assemble a representative case set and freeze it. Choose cases that reflect the variation you actually care about, then stop editing the set. Both prompts run against this exact set.
- Record the full configuration. Capture model and version, prompt and system-instruction version, examples and their order, temperature, top-p, output limits, structured-output mode, retrieval and reranking configuration, tool definitions and permissions, and the date and provider region.
- Run both variants under identical conditions. Same cases, same settings, same tool access, same number of attempts.
- Score with the same rubric. Where feasible, score without knowing which prompt produced which output; blind scoring removes the temptation to grade the variant you expect to win more kindly.
- Repeat and record the spread. One run may favour a variant by chance. Repeat enough to observe variability, especially for open-ended or agentic tasks.
- Report the result honestly. Give the average or median performance and the variability, not just the best-looking single run.
How many runs, and what to report
There is no universal number of repetitions that makes a comparison valid; the right count depends on how much the outputs vary. Open-ended and agentic tasks vary more than deterministic classification, so they need more runs before a difference means anything. The practical rule is to repeat until you can see the variability, then report it. A comparison that reports “Prompt B scored 0.8” without a spread is weaker than one that reports the median and the range across runs, because the first number hides whether the gap is stable or a single lucky draw.
An illustrative example: two prompts for ticket priority
Suppose you have two prompts that label support tickets as High, Medium, or Low. Prompt A is a terse instruction. Prompt B adds worked examples. You run each once: Prompt A scores 0.6, Prompt B scores 0.8. It is tempting to declare Prompt B the winner and move on.
That conclusion is only safe if nothing else differed. If Prompt B happened to be tested on easier tickets, if its outputs were graded with a looser metric, or if it received five attempts while Prompt A received one, any of those differences could account for the gap on its own — with the prompt wording having nothing to do with it. This scenario is illustrative, adapted from the framing in the CodeSignal lesson, not a reported experiment.
A controlled version of the same test looks like this: one frozen set of tickets, one rubric, the same number of attempts for each prompt, the same model version and sampling settings, and scoring done without the scorer knowing which prompt produced which label. Then the score difference, if any, has a defensible interpretation.
Confounds to check before you trust a winner
| Confound | What it does to the comparison |
|---|---|
| Different test cases | One prompt may simply have seen easier inputs. |
| Different scoring rubric or strictness | The gap reflects grading, not prompt quality. |
| Unequal number of attempts | More tries can look like better performance. |
| Changed sampling settings | Temperature, top-p, or output limits alter outputs independently of the prompt. |
| Changed retrieval or reranking | Adds a second variable; you can no longer attribute the result to the prompt. |
| Changed tool access or permissions | One variant has capabilities the other lacks. |
| Different model version or region | Provider-side differences can dominate prompt effects. |
If you find a confound after the fact, the honest move is to rerun the comparison with it removed, not to explain it away. Unrecorded settings turn results into anecdotes that cannot be reproduced or explained.
Local models and automation workflows
Local runtimes are often the easiest place to run a controlled comparison, because you set the sampling parameters yourself and you know exactly which model file and runtime version produced the output. Record the quantization, runtime version, and context length alongside the usual settings, and if your runtime supports a fixed seed, record that too. Holding these constant between the two prompt runs is what makes the comparison meaningful.
The same discipline applies to automated workflows. If a prompt is embedded in a WordPress or automation pipeline, treat each variant as a versioned template, keep the frozen case set in a file rather than retyping inputs, and log the run configuration with every output. That log is what lets you answer the question months later when someone asks why Prompt B was chosen. Note also that retrieval is part of the comparison: if you change the retrieval configuration between the two prompt runs, you have introduced a second variable and the result can no longer be attributed to the prompt.
Limitations a fair comparison still has
A clean comparison answers the question it was designed to answer, and little more. A frozen case set is representative of itself, not of production traffic, so a win on your cases is evidence — not a guarantee — that the prompt will behave the same way on new inputs. Time-sensitive data can drift between runs. Subjective rubrics introduce scorer judgement, which is why blind scoring and, where possible, a second scorer help.
There is also a deliberate choice to make when a model needs a different prompt format. You can test identical prompting across models, or the best reasonable configuration for each model — but those are different questions, and the adaptation should be documented either way. If you change the format for one variant and not the other, you have changed more than the prompt.
Next action
Pick one decision you actually need to make, write it as a comparison question, and freeze a case set that reflects the inputs you care about. Run both prompts under identical conditions, log the full configuration, and report the median result and its spread rather than a single run. If you hit a confound you cannot remove, say so in the write-up — a comparison with a known limitation is still more useful than a score with no provenance.
If you are working through a comparison of your own and something does not line up, share your goal, environment, test set size, and observed results in the community so others can help check for confounds.
FAQ
Can I compare two prompts with just one run each?
No. A single run may favour one variant by chance. Repeat the cases enough to observe variability, then report the average or median performance and the spread rather than one score.
Is using the same input text enough to make a comparison fair?
No. You should also hold allowed sources, expected behaviour, the scoring rubric, the number of repetitions, tool access, and time-sensitive data steady, and record the full run configuration.
What should I do if the two prompts need different formats?
Document the adaptation and state whether you are testing identical prompting or the best reasonable configuration for each model. Those are different comparison questions, and changing format for only one variant adds a second variable.
How do I know whether the score difference is real?
First check for confounds such as unequal attempts, different rubrics, or changed sampling settings. Then check whether the gap persists across repeated runs and is larger than the run-to-run variability.