Testing several variants: control false discoveries

Count the claims you intend to make

Testing three variants against one control is three comparisons if each variant can be declared better than the control. It becomes six pairwise comparisons if you also want to distinguish every pair of variants. Define that family before looking at the table; selecting only the favorable comparison afterward understates the opportunity for error.

The same issue appears when a team searches many metrics, segments or stopping dates for a favorable result. More ideas can be useful for exploration, but the resulting leads need an evidence process that reflects how they were selected.

A small decision example

Consider a fictional experiment with control A and variants B, C and D. The only planned confirmatory claims compare each variant with A on one primary binary outcome. Assume the individual tests are valid for the data and the reported p-values come from the predeclared method.

For a chosen family-level allowance of 0.05, a Bonferroni allocation gives each of the three comparisons approximately 0.0167. B would pass an isolated 0.05 comparison but not this family rule. C passes the illustrative family threshold. That still leaves effect size, uncertainty, data quality and business value to review.

NIST explains the Bonferroni inequality for simultaneous statements: the overall error can be bounded through the sum of individual error allowances. Its worked chapter concerns contrasts of means; the family-error principle here does not make a mean-based test appropriate for binary outcomes. [NIST: Bonferroni’s method]

Know what the adjustment does and does not assume

A Bonferroni bound does not require the comparisons to be independent, which is useful when they share a control. It can be conservative. Other multiplicity procedures may be appropriate, but they require a specified method and correct implementation. Choose the design for the question rather than selecting the adjustment that makes a favored result pass.

For intuition only, if ten valid null comparisons were independent and each had a 5% false-positive probability, the chance of at least one false positive would be 1 − 0.95^10, approximately 40.1%. Shared-control comparisons need not be independent, so do not present 40.1% as the actual probability for every ten-variant experiment.

No adjustment repairs broken assignment, an outcome collected differently by group, or a denominator changed after seeing results. Multiplicity control belongs beside those checks, not in place of them.

Keep exploration productive without overstating it

• Declare the primary outcome and the comparison family in the experiment card.
• Choose whether the objective is one best candidate, several acceptable alternatives or a specific planned contrast.
• Report the observed effect and uncertainty with the chosen multiplicity treatment.
• Label post hoc segment findings as exploratory and retain the search context.
• Use a clean planned confirmation when a selected exploratory candidate will drive a costly decision.

If the team can afford only one well-powered contrast, reduce the variant set or screen ideas through nonexperimental quality checks before launch. A large family with tiny groups can produce a colorful ranking without answering the commercial question.

Preserve the family in the final report

I would save all planned contrasts, including unfavorable results, with the final choice. If C improves conversion by a commercially negligible amount or breaches an error guardrail, a passing statistical comparison is not sufficient for rollout. The report should explain both the evidence rule and the operating decision.

• Develop distinct test ideas before selecting variants
• Review the experiment workflow
• Check a suitable single binary contrast

Sources

AI-assisted writing and an AI-generated editorial cover. Source links and checked dates are listed below; illustrative examples are labeled in the text.

• NIST: Bonferroni’s method — checked 2026-10-18