Statistical concepts
Statistical analysis methods overview
Optimizely Feature Experimentation Optimizely Analytics Optimizely Feature Experimentation offers the following statistical analysis methods for running A/B tests Frequentist (Fixed Horizon) – Uses a predetermined sample size and fixed plan. Bayesian statistics – Focuses on updating your beliefs about a hypothesis as new evidence becomes available.
Configure a Frequentist (Fixed Horizon) A/B test
Optimizely Feature Experimentation Optimizely Analytics Follow these steps to configure a Frequentist (Fixed Horizon) test in Optimizely Feature Experimentation. This configuration lets you run an A/B test with a predetermined sample size and a strict analysis plan.
Frequentist (Fixed Horizon) statistics
Fixed Horizon is a frequentist statistical method used to run traditional A/B tests with a predetermined sample size. This approach relies on well-established statistical concepts such as p-values, minimum detectable effect (MDE), and variance to determine whether observed differences between variations are meaningful.
Configure a Bayesian A/B test
Optimizely Feature Experimentation Optimizely Analytics Follow these steps to configure a Bayesian A/B test in Optimizely Feature Experimentation. This configuration lets you run an A/B test without a predetermined sample size and a strict analysis plan.
Bayesian statistics
Bayesian statistics provides a powerful framework for analyzing experiment data, offering a distinct approach compared to traditional Frequentist (Fixed Horizon) statistics . In Optimizely, this lets you incorporate prior knowledge (the current version of Bayesian statistics uses uninformed priors), gain intuitive insights, and make data-driven decisions with a focus on the probability of hypotheses.
Statistical significance
Optimizely Web Experimentation Optimizely Performance Edge Optimizely Personalization Optimizely Feature Experimentation Optimizely Full Stack (Legacy) Statistical significance https //www. optimizely.
Confidence intervals and improvement intervals
Optimizely Web Experimentation Optimizely Performance Edge Optimizely Personalization Optimizely Feature Experimentation Optimizely Full Stack (Legacy) Statistical significance https //www. optimizely.
False discovery rate control
Optimizely Web Experimentation Optimizely Performance Edge Optimizely Feature Experimentation Optimizely Full Stack (Legacy) Every experiment has a chance of reporting a false positive (reporting a conclusive result between two variations when there is actually no underlying difference in behavior between them). You can calculate an experiment's error rate as 100 - \[statistical significance] .
Optimizely's results page does not sample from your data
Optimizely Web Experimentation Optimizely Performance Edge Optimizely Feature Experimentation Optimizely Full Stack (Legacy) Sampling https //en. wikipedia.
How and why statistical significance changes over time in Optimizely Experimentation
Optimizely Web Experimentation Optimizely Personalization Optimizely Performance Edge Optimizely Feature Experimentation Optimizely Full Stack (Legacy) Optimizely's Stats Engine uses sequential experimentation https //www. optimizely.
How Optimizely Experimentation handles outliers
Optimizely Web Experimentation Optimizely Personalization Optimizely Performance Edge Optimizely Feature Experimentation Optimizely Full Stack (Legacy) In statistics, an outlier https //en. wikipedia.
Why Stats Engine results sometimes differ from traditional statistics results
Optimizely Web Experimentation Optimizely Personalization Optimizely Performance Edge Optimizely Feature Experimentation Optimizely Full Stack (Legacy) Sometimes, Optimizely declares a winning variation in a situation where a traditional t-test would fail to find any statistically significant difference between it and the other variations. This is because Optimizely Stats Engine uses an approach that differs from those used in classical statistics-based models, which is simultaneously more conservative in declaring a winner and less likely to reverse that declaration as more data accumulates.
Why is my experiment failing to reach statistical significance?
Optimizely Web Experimentation Optimizely Personalization Optimizely Performance Edge Optimizely Feature Experimentation Optimizely Full Stack (Legacy) If you have been running A/B tests, you have probably wondered why your experiment did not reach statistical significance. Statistical significance https //www.
Calculate the statistical likelihoods of variations using Stats Engine
Optimizely Web Experimentation Optimizely Personalization Optimizely Performance Edge Optimizely Feature Experimentation Optimizely Full Stack (Legacy) When you run experiments, Optimizely determines the statistical likelihood of each variation actually leading to more conversions https //support. optimizely.
Stats Engine resources
Optimizely Web Experimentation Optimizely Performance Edge Optimizely Personalization Optimizely Feature Experimentation Optimizely Full Stack (Legacy) Statistics are a core part of Optimizely. Learning how the Stats Engine processes your customer data lets you get more meaningful results from your experiments, turn those results into action, and build an effective culture of experimentation in your company.