What is an A/B Test?

An A/B test (split test) is a controlled experiment in which two versions of an element are shown at the same time to two comparable groups. The version that performs better on the chosen metric wins. A/B tests are the scientific basis of CRO and of optimisation in SEA and social media marketing.

How does an A/B test work?

You define a hypothesis (e.g. “A red CTA button converts better than a blue one”), create two variants (A and B), split the test traffic evenly, wait until you reach statistical significance and compare the results on the chosen metric (conversion rate, CPA). The winner becomes the new standard.

What can you A/B test?

In Meta Ads: ad variants, headlines and images. On landing pages: headlines, images, forms and CTAs. In email marketing: subject lines, sender names and layouts.

When is an A/B test statistically reliable?

At least 100 conversions per variant for conversion-related tests and a confidence threshold of 95 per cent. Heatmaps help you decide what is most worth testing.

How many variants can I test at once?

In a classic A/B test you test two variants. Testing more variants at once is called a multivariate test.

How long should an A/B test run?

At least one full weekly cycle (7 days) to rule out weekday effects. More at developers.google.com/search.