Read an A/B test with delayed conversions
Compare equally mature exposure cohorts, preserve pending outcomes and choose a predeclared conversion window instead of reading fresh groups as failures.
Give both groups the same chance to convert
An exposure today has had less time to convert than one recorded last week. In a sales-assisted funnel, that difference can dominate a live A/B result. Choose a follow-up window and compare equally mature exposure cohorts before calling unresolved outcomes nonconversions.
The window should match the decision. A confirmed inquiry within seven days and a paying customer within thirty days are separate outcomes. The example uses seven days for a synthetic demonstration; it is not a recommended universal window for B2B sales.
Separate mature, pending and observed counts
A fictional test assigns 500 independent units to each group. At the review time, A has 400 units with a complete seven-day follow-up, while B has only 250 because its exposure volume arrived later. Each group also contains more recent units whose final seven-day outcome is pending.
| Measure | A | B |
|---|---|---|
| All assigned exposed units | 500 | 500 |
| Complete seven-day follow-up | 400 | 250 |
| Pending seven-day follow-up | 100 | 250 |
| Conversions in the mature subset | 20 | 15 |
| Mature-subset observed rate | 5% | 6% |
Dividing 20 by all 500 units in A and 15 by all 500 in B would produce 4% versus 3% while treating pending observations as completed failures. The mature subsets instead show 5% versus 6%. Neither calculation alone validates a causal conclusion: the mature cohorts must also be comparable in exposure time and eligibility.
For a cleaner planned snapshot, select a common exposure-date range that has completed follow-up for both groups. If the same first 250 eligible units per group were not exposed over comparable periods, choosing them by count afterward can introduce another selection problem. Use the predefined cohort rule.
Write the event and timing rule together
Eligible exposure: successful assigned landing-page render
Outcome: at least one durably saved inquiry within 7 × 24 hours
Clock origin: that unit’s first eligible exposure
Snapshot rule: include exposure cohorts with complete follow-up
Late after-window outcomes: retain separately, outside primary flag
Pending: not yet eligible for the mature snapshotMicrosoft’s experiment-design guidance emphasizes the unit and pre-experiment plan. A delayed-outcome contract adds a clock and follow-up rule to that design, rather than deciding maturity after the favorable group appears. [Microsoft Research: Pre-experiment stage]
Keep the original event time and arrival time where the system supports them. A late-arriving event may describe a timely conversion that was delayed in processing. That is different from a customer who converted after the predefined window. Reconcile the distinction before editing the outcome flag.
Choose a method for the unresolved journey
If many outcomes remain pending, wait for the planned mature comparison or use a method explicitly designed for time-to-event data and its assumptions. A simple binary calculator is not a survival-analysis tool. Submitting the current incomplete totals does not make censoring disappear.
Review how a longer window changes the business question and the cost of obtaining evidence. A thirty-day purchase outcome may be more relevant than a same-day inquiry but slower to resolve. Do not keep extending the window solely because the original comparison failed to favor the proposed variant.
- Record exposure dates, clock origin, event time and report cutoff.
- Use the same eligible outcome and follow-up rule in both groups.
- Show pending and complete units separately.
- Preserve late and unresolved events for diagnosis without silently changing the primary outcome.
Show what can be concluded today
I would report the number of mature units, the pending share and the scheduled final observation cutoff alongside the interim rates. If a delay is in the data pipeline rather than customer behavior, name that separately. A useful interim report can explain why a decision is still open without inventing a winner.
Sources
AI-assisted writing and an AI-generated editorial cover. Source links and checked dates are listed below; illustrative examples are labeled in the text.
- Microsoft Research: Pre-experiment stage — checked 2026-10-18