Check sample ratio mismatch before reading a test
Compare expected and observed assignment counts, reproduce a sample-ratio check and investigate a suspicious split before interpreting conversion lift.
Check allocation before conversion
If a test plans a 50:50 split, the observed counts will not be exactly equal every time. The question is whether the imbalance is unusually large under the planned assignment model. That is a different question from whether the conversion rates differ.
Microsoft Research describes sample ratio mismatch as an experiment-quality problem requiring diagnosis before interpreting effects. It can arise at several stages, so a mismatch flag should begin an investigation rather than name a cause. [Microsoft Research: Diagnosing Sample Ratio Mismatch]
A reproducible two-group check
A fictional experiment has 10,000 eligible independent assigned units. It planned equal allocation but records 5,200 units in A and 4,800 in B. The expected count is 5,000 in each group. Use assignment counts, not the number of converting users, in this check.
| Group | Planned share | Expected count | Observed count |
|---|---|---|---|
| A | 50% | 5,000 | 5,200 |
| B | 50% | 5,000 | 4,800 |
Chi-square = (5,200 − 5,000)^2 / 5,000
+ (4,800 − 5,000)^2 / 5,000
= 8 + 8 = 16
Degrees of freedom = 2 groups − 1 = 1NIST documents the chi-square goodness-of-fit statistic as the sum of squared observed-minus-expected differences divided by expected counts. With one degree of freedom, the example statistic has an approximate tail probability of 0.000063 under the specified model. [NIST: Chi-Square Goodness-of-Fit Test]
That small probability makes the split suspicious under the planned model. Choose the alert threshold and monitoring approach before the test rather than borrowing a universal cutoff from this example. Repeated monitoring, many experiments and intentional changes to allocation need their own treatment.
Use the planned population, including exclusions
Assignment and exposure are separate stages. A server can assign a visitor to B before a script fails to log exposure. An exposure-only report could then have an imbalance even if assignment was working. Compare counts at each stage rather than substituting one for another.
- Start with eligible assignment records and the actual allocation configuration.
- Separate assigned, successfully exposed and outcome-observed units.
- Apply only the population exclusions specified in advance or justified by a recorded defect.
- Inspect missing counts by browser, device, geography, entry route and instrumentation version.
- Preserve duplicate and crossover evidence instead of deleting it until the totals match.
If allocation intentionally changed from 50:50 to 80:20 midway, calculate expected counts using the correct time-specific assignment probabilities. Testing the entire run against a single 50:50 total would create a false alarm. The same applies to eligibility filters that were changed during the run.
Find the earliest point where counts diverge
Compare the assignment service, page response, exposure event and reporting table for the same synthetic journeys. A redirect that removes one variant, an exception before logging, a stale cache or a join that drops unmatched IDs can all affect observed counts. The check itself cannot distinguish them.
Do not trim records from the larger group to force equal counts. That would conceal the symptom and can introduce selection bias. Document the earliest affected boundary, the time range and whether a clean re-run is required. A repaired dashboard does not retroactively repair missing exposure evidence.
The significance calculator compares binary conversion outcomes; it is not a sample-ratio checker. Clear allocation and data-quality concerns first, then use an outcome analysis suited to the surviving valid experiment.
Sources
AI-assisted writing and an AI-generated editorial cover. Source links and checked dates are listed below; illustrative examples are labeled in the text.
- Microsoft Research: Diagnosing Sample Ratio Mismatch — checked 2026-10-18
- NIST: Chi-Square Goodness-of-Fit Test — checked 2026-10-18