Statistical analysis methods overview
- Optimizely Feature Experimentation
- Optimizely Analytics
Optimizely Feature Experimentation offers the following statistical analysis methods for running A/B tests:
- Frequentist (Fixed Horizon) – Uses a predetermined sample size and fixed plan.
- Bayesian statistics – Focuses on updating your beliefs about a hypothesis as new evidence becomes available.
- Sequential (Optimizely Stats Engine) – Uses sequential testing that lets you monitor results continuously.
Choosing the right method depends on your traffic levels, experiment goals, and how flexible you need to be while the experiment runs. The following sections explain each method, compare their strengths and limitations, and help you decide which fits your needs.
Frequentist (Fixed Horizon)
What is meant by Frequentist (Fixed Horizon)?
Frequentist (Fixed Horizon) is a traditional statistical framework based on the idea that probabilities represent the long-term frequency of events. In this approach, you make decisions using fixed rules and predetermined sample sizes. Frequentist (Fixed Horizon) methods rely on statistical concepts like p-values, confidence intervals, and statistical significance to determine whether an observed result is likely due to chance. Fixed horizon testing follows this framework by requiring you to define the sample size and analysis plan before the experiment begins, and only evaluating results after all data is collected.
Frequentist (Fixed Horizon) experiments
Frequentist (Fixed Horizon) experiments require you to calculate the sample size before you start, and you can only view results after the experiment reaches the required sample size and minimum duration.
Frequentist (Fixed Horizon) is strict and controlled. Remember the following:
- You cannot peek mid-test – You cannot check full results before you reach the required conditions (sample size and duration). Checking experiment results before you reach the conditions can drastically increase the chances of false positives.
- You can only analyze results once – You can review outcomes only after you meet the experiment conditions (sample size and duration).
- You can support rigorous testing – You can maintain strict statistical standards using a traditional A/B testing process.
Bayesian
What is meant by Bayesian?
Bayesian statistics is a framework in which probabilities represent degrees of belief that are updated as new evidence becomes available. It provides the probability of a given variation being better than a baseline.
In simpler terms, if you say, "I believe there is an 80% chance Variation A is better than the baseline", you are expressing a degree of belief. The Bayesian statistical framework then systematically updates these degrees of belief as the A/B test progresses and you collect data.
Bayesian experiments
- Let you monitor results continuously during the experiment.
- Update results as new data comes in.
- Are closer to how you typically understand probability. For example, Bayesian says, "there is a 90% chance Variation A is better than Variation B, offering a more straightforward understanding compared to the p-values from Frequentist (Fixed Horizon) and Sequential methods.
Sequential (Optimizely Stats Engine)
What is meant by sequential?
There are different types of sequential testing, but overall, it is a statistical method that lets you analyze experiment results continuously as data is collected, rather than waiting until the end of the experiment. You can monitor performance in real time, make decisions earlier, and stop an experiment once there is enough evidence to declare a winner or make changes. Sequential testing at Optimizely is based on Optimizely's proprietary Stats Engine. See the links in the Sequential section for more information.
Sequential experiments
Using the sequential method in A/B tests lets you do the following:
- Monitor results continuously during the experiment. The results are always valid because Optimizely controls the false discovery rate.
- Stop the experiment early if you have reached a clear winner.
- Skip doing pre-experiment work, such as calculating the sample size before launching the experiment.
Sequential testing provides flexibility and speed but uses more advanced statistical methods to maintain accuracy.
The difference between Frequentist (Fixed Horizon), Bayesian, and sequential analogy
Think of the difference between the statistical methods like checking a cake while baking:
- Frequentist (Fixed Horizon) testing – You must first decide how long to bake the cake before putting it in the oven. After putting it in the oven, you must wait for the timer to go off before you look inside, even if the cake might have been done earlier. If you take the cake out early, even if it looks ready on the outside, it may still be raw inside.
- Bayesian testing – You can put the cake in the oven with an initial idea of how long it might take. You can open the oven door at any time to check its progress, and each check updates your belief about whether it is done. If it looks done, you can take it out, and your confidence in its readiness is based on all the information gathered so far.
- Sequential testing – You can put the cake in the oven without deciding how long it needs to bake. You can open the oven door at any time to see if the cake is ready without ruining it. If it looks done, it is ready.
When to use each method
Use the following table to compare how Frequentist (Fixed Horizon), Bayesian, and sequential methods differ in planning, peeking, speed, and decision-making:
Feature | Frequentist (Fixed Horizon) | Bayesian | Sequential |
|---|---|---|---|
Sample size calculation | Required before starting | Not required | Not required |
Peeking at results mid-test | Not allowed | Allowed | Allowed |
Early stopping | Not allowed | Allowed | Allowed |
Best for | Fixed duration. You can roughly estimate the experiment's duration to reach the target sample size. | Flexibility, intuitive results. | Flexibility in results analysis, but at the same "Frequentist" way of interpreting results. |
When to use Frequentist (Fixed Horizon)
Use Frequentist (Fixed Horizon) if the following are true:
- You have strong statistical expertise – Your team has a collaborator who understands traditional A/B testing and can design and analyze a Frequentist (Fixed Horizon) experiment.
- You can accurately estimate the sample size – You must be able to estimate the required sample size before starting your experiment. If the estimation is inaccurate, your test may be underpowered, or the required sample size may be bigger than necessary.
- You expect small, precise changes – You are testing subtle optimizations where tight control matters.
- You can wait until the experiment ends to review results – You do not need to monitor or adjust results mid-test.
Important
Viewing results before the experiment meets the required sample size and duration can invalidate your experiment and increase false positives.
When to use Bayesian
Use Bayesian if the following are true:
- You need flexibility to check results anytime – You want to monitor results continuously throughout the experiment and make decisions as soon as sufficient evidence is gathered, without invalidating the test.
- You prefer intuitive interpretation – You want results expressed as direct probabilities (for example, "there is a 90% chance that Variant B is better than Variant A"), which can be easier to communicate to stakeholders.
- You want to make early decisions or reallocate traffic mid-test – You plan to act on results before the experiment ends, potentially leading to faster iterations and optimizations.
When to use sequential
Use sequential if the following are true:
- You need flexibility to check results anytime – You want to monitor results continuously throughout the experiment.
- You make early decisions or reallocate traffic mid-test – You plan to act on results before the experiment ends.
Next steps
See the following documentation to learn more about the different statistical methods Optimizely employs.
Frequentist (Fixed Horizon)
- Configure a Frequentist (Fixed Horizon) A/B test – Follow the step-by-step guide to configure a Frequentist (Fixed Horizon) A/B test.
- Frequentist (Fixed Horizon) statistics – Learn the mathematical and statistical concepts behind Frequentist (Fixed Horizon) testing.
Bayesian
- Configure a Bayesian A/B test – Follow the step-by-step guide to configure a Bayesian A/B test.
- Bayesian statistics – Learn the mathematical and statistical concepts behind Bayesian testing.
Sequential
- Stats Engine resources — Lists documentation showing how Stats Engine works, including comparisons with classical statistical methods.
- Statistical concepts — Defines a deeper explanation of confidence intervals, improvement intervals, false discovery rate, outliers, and so on.
- How and why statistical significance changes over time in Stats Engine — Explains how evidence builds and results evolve during an experiment.
- Calculate the statistical likelihoods of variations using Stats Engine — Helps you understand what variation lifts and confidence levels mean.
- Why Stats Engine results sometimes differ from classical statistics — Clarifies the differences you may see if you are used to traditional statistical methods.