How Many Visitors Do You Need for an A/B Test?
The four inputs that set a test's sample size, and a worked example you can repeat.
Key takeaways
- 01Sample size depends on the baseline conversion rate, the smallest change worth detecting, the significance level and the power.
- 02A small expected lift on a low-conversion page needs many more visitors than most teams expect.
- 03Decide the sample size and the stop rule before the test starts, not after you see the result.
- 04A test that ends early because it looks good often overstates the result.
Why sample size matters
An A/B test needs enough visitors in each variant to tell a real difference from random variation. Too few visitors and the test cannot detect a real change; too many and you waste time. Work out the sample size before the test starts.
The four inputs
- Baseline conversion rate: the current rate on the page you are testing.
- Minimum detectable effect: the smallest relative change that would be worth acting on.
- Significance level: the chance of a false positive you accept, commonly 5%.
- Power: the chance of detecting a real effect of that size, commonly 80%.
A worked example
Suppose the baseline conversion rate is 3%, and you want to detect a 10% relative lift (from 3% to 3.3%). At 5% significance and 80% power, the test needs several tens of thousands of visitors per variant. Lower baselines and smaller lifts push the number up quickly.
Use a sample size calculator for the exact figure, and keep the inputs in your test log so the decision can be checked later.
Set the stop rule in advance
Decide the sample size and the end date before you start. Checking the result every day and stopping when it looks good inflates false positives. If you must check early, use a method designed for repeated looks, and say so in the test plan.
When a test cannot reach the sample size
Low-traffic pages may never reach a reliable sample size for small changes. Options include testing bigger changes, testing on a higher-traffic page in the same funnel, or using research such as user sessions and surveys to decide, without claiming a statistical result.
Frequently asked questions
Is 95% significance always enough? It is a common default, not a guarantee. The right threshold depends on the cost of a wrong decision.
Can I test several things at once? You can, but each extra test raises the chance of a false positive. Adjust the plan or test one change at a time.
