Measure a win-back campaign with a holdout
Design a win-back holdout with preassigned eligible groups, matched observation windows and contribution after incentives, using a worked illustrative comparison.
What does a holdout help you estimate?
A holdout asks what would have happened without the proposed intervention under the test conditions. Some dormant customers return on their own. Counting every purchase after a win-back message as recovered by the campaign overstates its contribution. Preassignment gives the comparison a defined baseline, provided the groups and execution support the design.
NIST’s guidance distinguishes large-sample proportion comparisons from small-sample treatment. A reactivation comparison needs the right denominator, assumptions and test choice. A statistically significant difference, if established, would still not prove that the incentive produces worthwhile contribution. [Comparing two proportions]
A preassigned test record
| Field | Chosen rule |
|---|---|
| Eligibility | Previously paying customers inactive under the declared business definition |
| Assignment | Random treatment or holdout before contact |
| Analysis | Assigned-group denominator, with exclusions defined beforehand |
| Window | Same start and end for both groups; allow outcomes to mature |
| Treatment | One declared message and incentive |
| Contamination | Record other campaigns and cross-group exposure |
| Primary outcome | Accepted reactivation under a fixed definition |
| Economic outcome | Contribution after variable costs and incentive expense |
Do not remove treatment recipients who did not open the email and compare the remainder with the full holdout. That changes the assigned-group comparison into a selected subgroup. Keep delivery failures visible and decide their treatment in advance. Exploratory subgroup analysis can be useful, but label it separately.
A complete illustrative USD comparison
Suppose 500 eligible customers are assigned to each group. Treatment produces 45 reactivations and holdout produces 30 over the same completed window. The observed rates are 9% and 6%, a difference of three percentage points. Applied to the 500-person treatment group, that is 15 more reactivations than the holdout rate would imply.
| Item | Calculation | Result |
|---|---|---|
| Extra reactivations at observed rate difference | 500 × (0.09 − 0.06) | 15 |
| Contribution before incentive | 15 × $40 | $600 |
| Incentive cost across treatment reactivations | 45 × $10 | $450 |
| Campaign execution cost | Declared included amount | $100 |
| Observed contribution difference after these costs | 600 − 450 − 100 | $50 |
The incentive cost applies to all 45 treatment reactivations in this example, not only the estimated 15 additional ones. Some people who would have returned anyway also receive the offer. The $50 result is an illustrative point estimate under the chosen boundaries, not a guaranteed incremental profit or a statistically established win.
What could invalidate the interpretation?
- Groups were assigned after outcomes or selected by recent engagement.
- Other campaigns reached one group differently.
- The observation window is incomplete or differs between groups.
- Customers share offers across groups.
- Repeated testing or early stopping changes the error rate without a declared method.
- Contribution excludes a material cost or ignores later returns.
Use an appropriate statistical analysis for the design and sample. Do not run a two-mean test on binary conversion outcomes simply because a spreadsheet offers it. Preserve counts and uncertainty alongside the business calculation. If the sample cannot distinguish plausible outcomes, record that limit and avoid declaring a winner.
Decide using evidence and economics together
The final record should contain eligibility, assignment, exposure, outcomes, costs and the decision rule. This makes the result inspectable and prevents a later report from turning every treatment purchase into campaign-caused revenue.
Sources
AI-assisted writing and an AI-generated editorial cover. Source links and checked dates are listed below; illustrative examples are labeled in the text.
- Comparing two proportions — checked 2026-10-18