A/B Test Significance Calculator
Compare two independent conversion rates with an approximate two-sided two-proportion z-test.
Your numbers
Example values are prefilled. Use the same reporting period for all inputs.
Control conversion rate
5%
Variant conversion rate
6%
Relative lift
20%
Z statistic (variant minus control)
3.1
Two-sided p-value as a percent
0.19%
5% corresponds to p = 0.05; this is not the probability a variant wins
Approximate 95% difference lower bound (pp)
0.37
Approximate 95% difference upper bound (pp)
1.63
The formula
a = Control converters / Control participants; b = Variant converters / Variant participants; p̂ = Total converters / Total participants; z = (b − a) / √[p̂(1 − p̂)(1/nA + 1/nB)]; two-sided p = 2Φ(−|z|). Difference interval = (b − a) ± 1.96√[a(1 − a)/nA + b(1 − b)/nB].
Worked example
500/10,000 control conversions versus 600/10,000 variant conversions is 5% versus 6%, a 20% relative lift. z ≈ 3.10 and two-sided p ≈ 0.19%; the approximate 95% difference interval is 0.37 to 1.63 percentage points.
Frequently asked
When is this test appropriate?
Use independent randomized groups, one binary outcome per participant, a fixed analysis plan, and a completed test. The normal approximation requires adequate counts. Repeated peeking, multiple comparisons, and non-random traffic can invalidate the interpretation.
Does p < 5% prove a business win?
No. It measures how unusual the observed difference would be under the equal-rate null model. Evaluate the effect size, uncertainty, costs, and guardrail metrics too. The interval uses an unpooled normal approximation.