Free tool
A/B test significance calculator
Your new variant is getting more orders than the old one. Before you ship it, check whether the difference is real or just traffic noise. Haven’t launched yet? Switch to “Plan a test” to see how many visitors you need.
What the calculator tells you
It compares the conversion rate of two versions of a page: the old one (A) and the new one (B). It tells you whether the difference between them is real or just normal traffic fluctuation. You need four numbers: the visitors and conversions of each version.
It runs a two-proportion z-test, the standard method behind most A/B testing tools. It works correctly when each visitor sees only one version, each version has at least 30 conversions, and the traffic is normal — not from a sale period.
How to read the result
p-value
How likely you are to see a difference at least this big if the two versions were in fact equally good. At p = 0.03, a gap like this would appear by chance only 3% of the time. Below 0.05 is the usual threshold.
Confidence level
The standard is 95%. Low-traffic stores sometimes use 90% so they don’t wait for months, but the risk of a false winner grows from 5% to 10%. For big changes such as checkout or pricing, 99% is worth the extra traffic.
The range
Shows where the real difference most likely sits. If the range includes zero, you can’t yet say B is better. A wide range means B is better, but you don’t know by how much.
How many visitors do you need?
The most common mistake is stopping the test too early. How many visitors you need depends on three things: your current conversion rate, the smallest lift you want to catch and the confidence level. The smaller the lift, the more traffic you need: halve it and you need about four times as many visitors.
Example for a store converting at 2%, at 95% confidence and 80% power:
| Lift you want to detect | Visitors per variant | At 5,000 visitors/week | At 1,000 visitors/week |
|---|
The takeaway: with low traffic, small tweaks can’t be tested. Test bold changes that could move conversion by 20% or more.
Mistakes that fake a win
- Stopping when it looks good. If you check daily and stop the first time p drops below 0.05, you catch luck, not a real effect. Decide the number of visitors in advance.
- Traffic doesn’t split evenly. If you set 50/50 and ended up at 53/47, something in the test setup is broken. The calculator warns you.
- Too few conversions. Below 30 conversions per variant, the calculation is no longer reliable.
- Testing in unusual periods. Black Friday, holidays or a big campaign change how people shop. What wins then can lose in a normal week.
- Too many variants at once. Every extra variant needs more traffic. With low traffic, test A against B.
Not sure what’s worth testing?
The conversion audit includes A/B test ideas, in the order worth running them. We redesign your first section for free.