What to do after an inconclusive A/B test
Choose whether to stop, repair, simplify or continue an A/B test when its evidence still allows both useful and unhelpful business outcomes.
Start with what remains possible
An inconclusive test can still change a decision. It may rule out the dramatic improvement that motivated an expensive feature while leaving a smaller benefit unresolved. Read the effect range against the cost of action rather than reducing the report to winner or no winner.
For a fictional example, assume the estimated conversion difference is +0.4 percentage points and the chosen valid analysis produces an interval from −0.7 to +1.5 points. Suppose the team needed at least +1.0 point to cover implementation and added service work. The result allows a loss, a small gain and a worthwhile gain. It has not resolved that launch question.
Repair an invalid experiment before buying more data
- Check that assignment, exposure and outcome counts use the planned unit.
- Investigate suspicious allocation imbalance and missing or duplicated events.
- Confirm equivalent observation time for outcomes in both groups.
- Review simultaneous changes, group crossover and the analysis population.
- Only then evaluate whether the remaining uncertainty is statistical or operational.
Microsoft’s pre-experiment guidance treats the randomization unit and validation checks as design decisions. The review sequence here applies that principle to a decision after a test; it is not a universal requirement to run an experiment for a particular number of days. [Microsoft Research: Pre-experiment stage]
If variant B loses outcome events after a redirect, collecting another month can make the biased comparison look more precise. Fix the event path, document which data were affected and decide whether a clean experiment is needed. Do not silently combine incompatible instrumentation versions.
Choose among four different next actions
| Evidence state | Practical question | Reasonable next action |
|---|---|---|
| Measurement defect | Can the existing estimate be trusted? | Repair and define a clean analysis period; do not call the old result a valid null. |
| Wide valid range | Could more evidence change a valuable decision? | Plan additional sampling or a new test with an explicit cost and analysis rule. |
| Range excludes a worthwhile gain | Is the remaining possible benefit worth implementation? | Stop this proposal or reduce its cost; do not claim mathematical equality. |
| Potential gain but sampling is too costly | Can a safer reversible choice serve the business? | Choose with a documented risk and economic rationale, preserving uncertainty. |
The last branch is common when a low-volume page cannot resolve a small effect cheaply. A reversible design choice can be made for accessibility, maintenance or clarity without claiming that the test proved a conversion benefit. Record the reason so that a product decision does not later become a fabricated test result.
Do not improvise a new stopping rule after looking
Before continuing, decide whether the existing analysis supports extension or whether a new planned test is required. Repeatedly checking a conventional fixed-horizon test and stopping at the first favorable result changes the evidence process. Use a justified sequential method if ongoing looks are part of the design, with someone able to validate its assumptions.
Define the smallest worthwhile effect in business units. An illustrative 0.2-point increase on 5,000 monthly eligible exposures represents ten additional expected conversions if it transfers. At USD 8 contribution each, that is USD 80 per month before change costs. Those chosen economics may make a long experiment unattractive even when a tiny positive effect exists.
A new sample-size plan should use a baseline, target effect, error criteria and experimental unit appropriate to the real setup. There is no universal number of conversions that makes every test conclusive.
Write what the test changed
I would finish the review with the trusted data range, unresolved effect range, practical threshold, next action and the evidence that could reopen the decision. Preserve any measurement defect separately from statistical uncertainty. The words “not significant” should never be the only explanation for stopping or continuing.
Sources
AI-assisted writing and an AI-generated editorial cover. Source links and checked dates are listed below; illustrative examples are labeled in the text.
- Microsoft Research: Pre-experiment stage — checked 2026-10-18