A/B Test Calculator

Analyze your A/B test results with confidence. Compute conversion rates, relative uplift, p-values, confidence intervals, and Bayesian posterior probabilities.

A Control Group
B Treatment Group
? Typical +25% lift: A(1000,100) B(1000,125)
? Small effect: A(2000,200) B(2000,220)
? Large effect: A(1500,150) B(1500,225)
? Low baseline: A(2000,40) B(2000,60)
? High baseline: A(1000,500) B(1000,540)
Privacy first: All computations run locally in your browser. No data is sent to any server.

Understanding A/B Test Analysis

A/B testing (also called split testing or bucket testing) is a controlled experiment where two variants — a control (A) and a treatment (B) — are compared to determine which performs better on a predefined metric, typically a conversion rate. The A/B test calculator applies rigorous statistical methods to help you decide whether observed differences are real or due to random chance.

Conversion Rate = Conversions / Visitors

Lift = CRB − CRA  |  Relative Lift = (CRB − CRA) / CRA

The z‑statistic for two proportions: z = (p̂B − p̂A) / √( p̂(1−p̂)(1/nA + 1/nB) )

Why Use a Dedicated A/B Test Calculator?

  • Statistical Rigor: Avoid common pitfalls like peeking at results or misinterpreting p‑values. The calculator provides clear significance thresholds.
  • Decision Confidence: Know whether your treatment truly outperforms the control, or if the lift is within the margin of random fluctuation.
  • Bayesian Alternative: For practitioners who prefer probabilistic interpretations, the Bayesian method yields the probability that B is better than A, plus expected loss.
  • Sample Size Planning: While this tool analyzes completed tests, understanding the required sample size is critical for experiment design — see our Sample Size Calculator.

Frequentist vs. Bayesian: Which Approach Should You Use?

The frequentist (classical) approach, implemented as a two‑proportion z‑test, computes a p‑value — the probability of observing a difference at least as extreme as the one in your data, assuming the null hypothesis (no difference) is true. If the p‑value is below your chosen significance level (e.g., 0.05), you reject the null and declare the result statistically significant. This method is widely used and well‑understood, but it does not directly answer: “What is the probability that B is actually better than A?”

The Bayesian approach addresses this directly. Using a Beta‑Binomial model with uninformative priors (Beta(1,1)), we compute the posterior distribution of each conversion rate. From these, we derive:

  • P(B > A | data): The probability that the treatment conversion rate exceeds the control, given the observed data.
  • Expected Loss: The expected regret if you choose the treatment but it turns out to be worse than the control, measured in absolute conversion points.

Bayesian results are more intuitive for decision‑making and naturally incorporate uncertainty. Both methods are provided here so you can compare and choose based on your analytical preference.

Interpreting Key Metrics

  • Conversion Rate (CR): The proportion of visitors who complete the desired action. A higher CR indicates better performance.
  • Absolute Lift: The raw difference in conversion rates (CRB − CRA). This tells you the percentage‑point improvement.
  • Relative Lift: The percentage increase relative to the control. For a control CR of 10% and treatment CR of 12%, the relative lift is 20%.
  • p‑value: Lower values suggest stronger evidence against the null. A p‑value < 0.05 is conventionally considered statistically significant.
  • Confidence Interval: The range within which the true lift is likely to fall, with a given confidence level (e.g., 95%). If the interval excludes zero, the result is statistically significant.
  • Number Needed to Treat (NNT): The number of visitors you need to treat with the treatment version to observe one additional conversion compared to the control. Lower NNT is better.

Common Pitfalls in A/B Testing

  • Early Stopping: Checking results before reaching the planned sample size inflates the false‑positive rate. Always pre‑determine sample size and duration.
  • Multiple Comparisons: Running many tests or segments increases the chance of finding a spurious significant result. Use correction methods (e.g., Bonferroni) if testing multiple variants.
  • Non‑Random Assignment: Ensure that visitors are randomly assigned to variants to avoid selection bias.
  • Seasonality and External Factors: Be aware of time‑based effects (holidays, promotions) that can confound results.
  • Over‑interpreting Small Samples: Small sample sizes produce wide confidence intervals and unstable estimates. Use power analysis to determine required sample size.

Real‑World Case Study: E‑Commerce Checkout Optimization

Case Study: Simplifying Checkout Flow

An online retailer ran an A/B test on their checkout page. The control (A) was the existing multi‑step checkout. The treatment (B) was a streamlined single‑page checkout. After 2,000 visitors per variant, the control had 120 conversions (6.0%), while the treatment had 156 conversions (7.8%).

Using this calculator, the absolute lift is 1.8 percentage points, and the relative lift is 30%. The p‑value is 0.025 (two‑tailed), which is below 0.05, indicating statistical significance. The 95% confidence interval for the lift is [0.3%, 3.3%], meaning we can be 95% confident that the true improvement is between 0.3 and 3.3 percentage points. The NNT is approximately 56 — meaning for every 56 visitors exposed to the treatment, one extra conversion is generated compared to the control.

Based on these results, the retailer confidently rolled out the single‑page checkout, resulting in a sustained 28% lift in conversion rate over the following quarter.

Frequently Asked Questions

Sample size depends on your baseline conversion rate, the minimum detectable effect you care about, your desired power (typically 80% or 90%), and your significance level (α). Use our Sample Size Calculator to determine the required number of visitors per variant before running your test.

The conventional threshold is 0.05 (5%). However, in high‑stakes decisions or when you have many tests, consider using 0.01 (1%) to reduce false positives. The calculator supports 80%, 90%, 95%, and 99% confidence levels.

A result is statistically significant if the p‑value is less than the chosen significance level (α). This indicates that the observed difference is unlikely to be due to random variation alone, providing evidence that the treatment has a real effect.

Use a two‑tailed test when you want to detect any difference (improvement or degradation) between A and B. Use a one‑tailed test when you only care about whether B is better than A, and you are not interested in detecting a negative effect. One‑tailed tests have slightly higher power for detecting improvements in one direction.

The Bayesian posterior probability, P(B > A | data), is the probability that the treatment conversion rate is higher than the control, given the observed data and prior assumptions. A probability above 0.95 is often considered strong evidence that B is better than A.

The confidence interval provides a range of plausible values for the true lift. If the interval does not contain zero, the result is statistically significant at the chosen confidence level. For example, a 95% CI of [0.5%, 2.5%] means we are 95% confident that the true lift lies between 0.5 and 2.5 percentage points.
References: Kohavi, R. et al. (2020). Trustworthy Online Controlled Experiments; Evan Miller's A/B Testing Resources; Wikipedia: A/B Testing; Optimizely Statistics Engine.

Built for experimenters, by experimenters. This A/B test calculator implements standard statistical tests (two‑proportion z‑test) and Bayesian Beta‑Binomial inference, validated against multiple reference implementations. The methodology follows established best practices from the Trustworthy Online Controlled Experiments book by Kohavi, Tang, and Xu, and adheres to the American Statistical Association's guidance on p‑value use. Reviewed by the GetZenQuery tech team, last updated July 2026.