Zero conversions in one test group: interpret the result
Report a sparse two-by-two conversion table, handle a zero baseline honestly and choose an exact method rather than forcing a large-sample calculator.
A zero cell is an observation, not a certainty
When one test group has no conversions, start with the exposure count. Zero conversions among 20 exposed units and zero among 20,000 units provide very different evidence. Neither should be rewritten as proof that the underlying experience can never convert.
Relative lift divides the rate difference by the baseline rate. With a zero baseline, that denominator is zero, so the relative result is undefined. Use the observed counts and percentage-point difference. An “infinite lift” label adds drama without providing a usable estimate.
Keep the complete two-by-two table
This fictional independent-unit test has one binary outcome per unit and a complete common observation window. Control A converts none of 20 units; B converts three of 20.
| Group | Converted | Did not convert | Total |
|---|---|---|---|
| A | 0 | 20 | 20 |
| B | 3 | 17 | 20 |
The observed rates are 0% and 15%, a difference of 15 percentage points. The point difference is large, but the evidence is sparse. The three successes may be commercially interesting enough to investigate while still leaving substantial uncertainty about the true rates.
NIST presents Fisher’s exact test for small independent binary samples, separately from its large-sample normal approximation. For this fixed-margins example, a conventional two-sided Fisher calculation gives approximately 0.231; it does not meet a prechosen 0.05 criterion. [NIST: Comparing two proportions]
The p-value is not the probability that the variant has no effect or that the result happened by chance. It describes a tail probability under the specified null model and method. The data do not resolve every plausible business effect simply because the point estimate looks striking.
What zero out of twenty still permits
Under an independent binomial model with a constant probability, a one-sided 95% upper bound after zero successes in 20 trials solves (1 − p)^20 = 0.05. The result is approximately 13.9%. This is a bound for A’s individual rate under those assumptions, not an interval for the between-group effect.
Upper bound for zero successes, one-sided 95%:
p_upper = 1 − 0.05^(1 / 20) ≈ 0.139
Observed rate remains 0 / 20 = 0Do not add one synthetic success to each group just to make a preferred calculator accept the input. Continuity corrections and Bayesian priors have legitimate uses in specified methods, but an improvised adjustment changes the analysis and should not be hidden.
Rule out a missing event before a missing conversion
- Verify that both groups could complete the action and reach confirmed receipt.
- Inspect whether A’s events are missing an identifier, blocked by a variant-specific error or absent from a reporting join.
- Confirm complete follow-up time and the same eligible unit.
- Choose a sparse-data method that matches the question.
- Compare the uncertainty with the cost and reversibility of the business decision.
A broken event path can create a zero that has nothing to do with customer behavior. More sampling will not repair it. If measurement is valid but exposure is limited, choose whether a new planned test or another form of evidence is worth the time. Avoid extending only until the first favorable result appears.
Use the right tool and report the limitation
The site’s A/B significance calculator uses a large-sample comparison and is not a Fisher exact calculator. Do not force this sparse example through it as though the methods were interchangeable. Keep the table, exact-method specification and operating decision together.
I would report the 0/20 and 3/20 counts, the undefined relative lift, the absolute observed difference and the remaining uncertainty. That preserves a potentially useful signal without promoting three outcomes into a guaranteed improvement.
Sources
AI-assisted writing and an AI-generated editorial cover. Source links and checked dates are listed below; illustrative examples are labeled in the text.
- NIST: Comparing two proportions — checked 2026-10-18