A/B Testing Sample Size Calculator

Calculate required sample size for reliable A/B tests with statistical power analysis. Determine optimal experiment size for conversion rate optimization.

Corrected Sample Size Formula (two-sample proportion test):

n = (Zα/2 + Zβ)² × [p₁(1-p₁) + p₂(1-p₂)] / (p₁ - p₂)²

For one-tailed tests: Use Zα instead of Zα/2

Standard A/B Test
High-Risk Change
Exploratory Test
Small Effect Size
Baseline Metrics
%
Current conversion rate before implementing changes (0.1% to 99.9%)
%
Minimum improvement you want to detect (relative increase, 1% to 1000%)
Statistical Parameters
%
Probability that results are not due to chance (1 - α)
%
Probability of detecting an effect if it exists (1 - β)
Two-tailed tests are more conservative and commonly used
Calculating...

Understanding A/B Test Sample Size

A/B testing (also known as split testing) compares two versions of a webpage, app, or marketing asset to determine which performs better. Proper sample size calculation is critical for obtaining statistically valid results.

Why Sample Size Matters:

  • Underpowered tests: Too small sample sizes may fail to detect real effects (Type II error)
  • False positives: Small samples increase risk of detecting effects that don't exist (Type I error)
  • Resource efficiency: Proper sizing ensures tests complete in reasonable timeframes
  • Business impact: Reliable results lead to better decisions and ROI

Key Statistical Concepts

Term Definition Typical Value
Statistical Power (1-β) Probability of detecting an effect when it truly exists 80-90%
Confidence Level (1-α) Probability that results are not due to chance 95%
Minimum Detectable Effect (MDE) Smallest improvement you want to detect 10-30% relative
Baseline Conversion Rate Current conversion rate before changes Varies by industry
Type I Error (α) False positive: detecting effect when none exists 5% (for 95% confidence)
Type II Error (β) False negative: failing to detect a real effect 10-20%

Factors Affecting Sample Size

1

Effect Size: Smaller effect sizes require larger sample sizes to detect. A 5% improvement requires much larger samples than a 50% improvement.

2

Baseline Conversion Rate: Lower baseline rates generally require larger samples. Detecting a 10% relative improvement from 2% to 2.2% requires more data than from 20% to 22%.

3

Statistical Power & Confidence: Higher confidence (e.g., 99% vs 95%) and higher power (e.g., 90% vs 80%) both increase required sample size.

Practical Considerations

  • Test Duration: Consider business cycles (weekly patterns, seasonal effects)
  • Traffic Volume: Ensure you have enough visitors to complete test in reasonable time
  • Multiple Comparisons: Testing multiple variations requires larger samples or corrections
  • Resource Constraints: Balance statistical rigor with practical limitations
  • Sequential Testing: Some methods allow for early stopping but require adjustments

Frequently Asked Questions

Tests with insufficient sample size are underpowered, meaning they have a high probability of:
  • Failing to detect real improvements (Type II error)
  • Producing inconclusive or misleading results
  • Wasting resources on tests that cannot provide clear answers
As a rule of thumb, never run A/B tests with fewer than 100 conversions per variation.

Choose MDE based on:
  • Business impact: What improvement would justify implementation costs?
  • Historical data: What improvements have you seen in past tests?
  • Practical constraints: Larger MDE requires smaller samples
  • Risk tolerance: Smaller MDE detects subtle effects but requires more data
For most marketing tests, 10-20% relative improvement is a reasonable MDE.

Two-tailed tests (default) are more conservative and recommended because they test for both improvement and deterioration. One-tailed tests only check for improvement in a specific direction but require stronger justification. Use one-tailed tests only when:
  • You're absolutely certain the change cannot harm performance
  • You're replicating a previous test with consistent directional results
  • You're testing a very low-risk change with strong theoretical backing
In practice, most A/B testing tools and practitioners use two-tailed tests.

Early stopping increases false positive rates unless you use special statistical methods like:
  • Sequential testing: Methods that adjust for multiple looks
  • Bayesian statistics: Approaches that handle continuous monitoring better
If using traditional frequentist statistics, avoid checking results frequently and never stop based on interim results unless:
  • Results are overwhelmingly significant (p < 0.001)
  • You have a predefined stopping rule with appropriate adjustments
  • The test has reached at least 50% of planned sample size

Lower baseline conversion rates generally require larger sample sizes for the same relative improvement. For example:
  • Improving from 2% to 2.4% (20% relative) requires more data than improving from 20% to 24%
  • This is because absolute variance is higher around 50% and lower near extremes
  • As a result, tests on low-conversion elements (like sign-up buttons) often need more traffic than tests on high-conversion elements
Always use actual baseline rates from your analytics rather than industry averages for accurate calculations.