A/B test

An A/B test is an experiment where two versions of something (ad, landing page, email) are shown to different random groups of the audience. Then it compares which version gave the best result for the selected metric (CTR, CR, revenue).

In my projects, an A/B test is the basis of any change. I'm not rolling out a "new version" of the landing page - I'm rolling out an A/B test between the old and the new. This saves you from failure.

Rules that I always follow: - Minimum statistical volume — 1000+ conversions for each option for reliable conclusions - Minimum period – 7+ days (takes into account daily cycles) - Only one variable at a time - otherwise it is impossible to understand what exactly worked - Fixed hypothesis before launch - “Option B will give more CR by 15%.” Without this - fishing expedition

Marketers often do A/B tests incorrectly: small sample size, premature conclusions, 2-day test with p=0.3 (“well, B seems to be better”). This is not an A/B test, it's self-deception.

A practical example from my practice - in the case -20% to the quarterly plan I just failed because I immediately rolled out a new segmentation for 100% of the budget instead of an A/B test for 30%. Lesson: Always the control group.