Use an A/A test to check the experiment pipeline
Design an A/A dry run that checks assignment, exposure, event delivery and reporting joins without treating one nonsignificant result as certification.
Use identical experiences to exercise different labels
An A/A test sends the same intended experience through two experiment labels. Its purpose is to expose pipeline problems before a product change makes them harder to separate from real behavior. It is a dry run of assignment, exposure, outcomes and reporting, not a conversion-improvement campaign.
Microsoft’s pre-experiment guidance includes validation practices before treatment analysis. The acceptance plan below is an original small-team checklist, rather than a claim that passing one A/A run reproduces Microsoft’s experimentation system. [Microsoft Research: Pre-experiment stage]
Write exact expectations for synthetic journeys
Begin with controlled QA cases whose intended events are known. For a synthetic unit assigned A, load the page once, navigate away and return, then submit one valid inquiry. The record should preserve the assignment and distinguish repeated exposures from the one binary conversion outcome.
| Stage | Expectation for a synthetic journey | Failure to investigate |
|---|---|---|
| Eligibility and assignment | One allowed label under the configured split | Fallback or cache places all units in one label |
| Repeat exposure | Stable label for the chosen unit | A new random draw occurs on reload |
| Outcome | One confirmed inquiry outcome after accepted receipt | Attempt and retry each count as a new binary success |
| Reporting join | Outcome joins to the same experiment unit/version | Missing IDs drop one label disproportionately |
| Failure path | Rejected or failed inquiry stays a failure state | Success event fires before durable receipt |
Keep synthetic traffic identified as technical and out of visitor business reports. Use approved test records and clean up only those records under the operating process. The A/A label alone does not prevent a test inquiry from reaching a real sales queue.
Then inspect the live allocation and metric behavior
For a permitted live dry run, record the planned allocation and compare observed assigned counts using a suitable sample-ratio check. Examine missing exposures, duplicate outcomes and group-specific failure rates. A 50:50 plan does not require exactly equal counts, but a suspicious split needs explanation.
Conversion rates can differ by chance even when the experiences are identical. A single p-value above the chosen threshold is therefore weak evidence about the whole pipeline. A p-value below it is not automatic proof of a bug either; investigate the design, data quality and the known false-positive behavior of the procedure.
Across repeated valid simulations or dry runs, evaluate whether the analysis behaves as intended under its assumptions. Preserve the full set of results rather than repeating until one reassuring run appears. If repeated runs share users or time periods, account for that dependency before treating them as independent checks.
Know which defects an A/A run can miss
A defect specific to a new variant may not exist in A/A. For example, an alternate form can omit the request ID while the shared baseline form works. A/A also cannot validate business acceptance rules that are missing from the source system. It should precede, not replace, variant-specific QA.
- Inspect the actual rendering and exposure boundary for each planned variant.
- Verify outcome semantics against a confirmed operating record.
- Test redirects, state loss and consent withdrawal where they affect the journey.
- Review instrumentation changes and cache behavior separately from statistical output.
Give the dry run an auditable completion condition
The acceptance note should state which journeys passed, which data-quality checks were reviewed, which analysis assumptions were exercised and what remains untested. Assign an owner and date for any unresolved defect. “A/A passed” without that scope can become a misleading guarantee.
I would carry the same QA matrix into the A/B release and rerun the rows affected by the new implementation. That keeps the dry run useful even if its conversion comparison happens to be entirely unremarkable.
Sources
AI-assisted writing and an AI-generated editorial cover. Source links and checked dates are listed below; illustrative examples are labeled in the text.
- Microsoft Research: Pre-experiment stage — checked 2026-10-18