Keep repeat visitors in one A/B test group
Specify a stable assignment unit for repeat visits, cookie loss and login transitions, with a concrete exposure contract and QA matrix.
Choose what receives the treatment
For a landing-page test, decide whether the treatment is assigned to an account, a browser installation, a session or another defined unit. Keep the outcome analysis consistent with that choice. A browser-level experiment should not claim to identify independent people across devices.
Microsoft’s pre-experiment guidance treats the randomization unit as a design choice. The practical contract below uses a browser-level unit for anonymous visits, with explicit limitations; it is not a claim that a cookie is a universal identity solution. [Microsoft Research: Pre-experiment stage]
A unit that sees both versions can carry an earlier offer or remembered price into the later experience. If the test concerns that journey, assigning on every page view changes the exposure being studied. Stable assignment is therefore part of the question, not merely a convenience for the analytics team.
A concrete assignment contract
experiment: inquiry-form-v3
unit: anonymous browser installation
assignment: stable within experiment version
first_exposure: first successful eligible render
repeat_exposure: retain original variant
analysis_outcome: one confirmed inquiry flag per unit in window
state_loss: new observable unit; continuity unknown
login_conflict: preserve active exposure; record for reviewStore an experiment version so that an intentional redesign does not reuse the meaning of an older assignment. Separate assignment from actual exposure: a request that is assigned B but fails before rendering has a different status from a unit that saw B. Decide the analysis population before the test.
The login rule needs to fit the product. A design might preserve the current assignment for the experiment and avoid account-level reassignment, or assign at account level from the outset. Either choice has limitations. A last-minute merge that rewrites past assignments makes the data hard to interpret.
Test the transitions a real visitor can take
| Journey | Expected behavior | Evidence to inspect |
|---|---|---|
| Reload and another page in the same browser | Same experiment version and variant | One stable unit and repeated exposure records without reassignment |
| Return later with state intact | Retain variant | No new random draw on session start |
| Clear anonymous state | Continuity is unknown | New observable unit; no claim of a new person |
| Sign in during an active journey | Apply the documented conflict policy | Original exposure preserved and conflict visible |
| Two devices with no shared account assignment | May be separate units | Do not deduplicate them into a claimed person after the fact |
Also test a cached page, a deep link, a failed assignment request and back/forward navigation. The fallback must be defined: a silent default variant followed by a later reassignment can expose one unit to both experiences. Log the failure state in permitted technical records without putting personal account data into analytics.
Carry identity limitations into the result
Repeat sessions from one assigned browser are not independent randomized units. Counting each session as a new exposure can make the uncertainty look smaller than it is. Aggregate the binary outcome at the chosen unit or use an analysis that explicitly handles the dependency.
Cross-device behavior, state loss and shared browsers limit interpretation. Measure what can be observed and retain a crossover or unresolved category when appropriate. Do not infer complete person-level continuity from a matching email, IP address or analytics device label.
- Document assignment unit, experiment version and persistence rule.
- Define first exposure and the outcome observation window.
- Specify state-loss and login conflicts before release.
- Review crossover, missing assignment and duplicate exposure separately.
Make the contract part of the landing-page brief
I would attach the assignment contract and synthetic cases to the landing-page specification before changing the offer. The significance calculation comes later. First establish that the groups describe the intended experiences for comparable units.
Sources
AI-assisted writing and an AI-generated editorial cover. Source links and checked dates are listed below; illustrative examples are labeled in the text.
- Microsoft Research: Pre-experiment stage — checked 2026-10-18