Statistical significance

Statistical significance is the probability that the observed result of an A/B test is not explained by random fluctuations in traffic. Standard threshold in marketing: p-value < 0.05, that is, the probability of mistakenly recognizing the result as real is less than 5%. This is called the “95% confidence level”.

Practically: if the test showed +12% conversion for option B with 95% significance, there is a 5% chance that this is just noise and there is no real effect. This is an acceptable risk for marketing decisions. For medical research, the threshold is stricter - p < 0.01 or p < 0.001.

The most common mistake in A/B testing is stopping the test too early. Example from practice: the test was launched on Monday, by Wednesday option B shows +25% conversion with 91% confidence, the test is stopped and the winner is announced. By Friday it turns out that the audience at the beginning of the week and at the end are different, and there is no real effect. Rule: Determine the minimum sample size before running the test (calculators are available), and do not touch the test until this sample is reached.

The second common sin is testing too many hypotheses at once. With 20 parallel tests with a 95% threshold, one “win” will turn out to be statistical noise simply according to the laws of probability. I stick to the rule: no more than 3-4 active tests simultaneously on the same traffic segment, and always with a Bonferroni correction for multiple testing.