Configure a Frequentist (Fixed Horizon) A/B test
- Optimizely Feature Experimentation
- Optimizely Analytics
Follow these steps to configure a Frequentist (Fixed Horizon) test in Optimizely Feature Experimentation. This configuration lets you run an A/B test with a predetermined sample size and a strict analysis plan.
Beta
Frequentist (Fixed Horizon) is in beta. Contact your Customer Success Manager for information.
Prerequisites
Before you begin, you must have the following:
- An Optimizely AnalyticsOptimizely Analytics account.
- A flag in Feature Experimentation. See Manage flagsManage flags.
Create a Frequentist (Fixed Horizon) A/B test rule
Create an A/B test rule
- Select a flag from your Feature Experimentation project.
- Create an A/B test rule in your desired environment. See Run A/B tests in Feature ExperimentationRun A/B tests in Feature Experimentation.
Configure your A/B test rule
- Configure your rule as normal.
- Complete the following for Stats Configuration:
- Select Frequentist (Fixed Horizon) for the Stats Configuration. This replaces the default Sequential (Optimizely Stats Engine) statistical methods with the Frequentist (Fixed Horizon) approach. See Statistical analysis methods overview for information on the differences between Sequential (Stats Engine) and Frequentist (Fixed Horizon) testing.
- (Optional) Update the Statistical Significance Level. The statistical significance level defaults to what is set in the project's Settings. See Change the statistical significance setting in Optimizely ExperimentationChange the statistical significance setting in Optimizely Experimentation for information. You should set the statistical significance level to 90% or 95%. The significance level must be between 70% and 99%.
Note
The significance level set here must match the value you use in the Frequentist (Fixed Horizon) Sample Size Calculator in the next step in this configuration.
Use the Frequentist (Fixed Horizon) Sample Size Calculator
Use the Sample Size Calculator to determine the number of visitors you need in each variation before you can analyze the results.
Click Sample Size Calculator and enter the following information:
Metric Type
Select the type of metric you want to analyze from the Metric Type drop-down list.
- Conversion Metrics – Binary metrics that track whether an event happened, such as a visitor converting or not. Examples include sign-ups or completed purchases.
- Numeric Metrics – Metrics that measure continuous values, such as revenue, items added to a cart, or total clicks per visitor.
Baseline Metric Value
Enter the current value of the metric you are testing, before any experiment changes.
The Baseline Metric Value is the starting point or current value of the metric you are testing before making any changes in your experiment. It represents how your metric is currently performing with no variations applied.
For example, if your sign-up rate is currently 5%, enter 5.
Metric Variance (Optional)
Note
For Numeric metrics only.
If you selected Numeric for the Metric Type, enter the Metric Variance.
Variance reflects the variation in metric values. A higher variance often means that more data may be needed to detect the treatment effect.
To calculate variance yourself, use the following formula:
\\[ \\text{sample variance} = \\frac{\\sum_{i=1}^(x_i - \\bar)^2} \\]
The following is a step-by-step example using example data to calculate variance for numeric metrics:
For example, your metric is "minutes spent in app per user" for five users, with values 4, 6, 5, 9, 6.
- Find the average of the data. \\[ \\frac{4 + 6 + 5 + 9 + 6} = 6 \\]
- For each data point, subtract the average to see how far it is from the mean. \\[ 4−6 = −2 \\] \\[ 6−6 = 0 \\] \\[ 5−6 = −1 \\] \\[ 9−6 = 3 \\] \\[ 6−6 = 0 \\]
- Square each of those differences. \\[ (−2)^2 = 4 \\] \\[ (0)^2 = 0 \\] \\[ (−1)^2 = 1 \\] \\[ (3)^2 = 9 \\] \\[ (0)^2 = 0 \\]
- Add up all the squared differences. \\[ 4 + 0 + 1 + 9 + 0 = 14 \\]
- Divide that total by (n − 1), where n is the number of data points. \\[ \\frac{5 - 1} = \\frac = 3.5 \\]
Minimum Detectable Effect (MDE)
Enter the Minimum Detectable Effect. The MDE is the smallest change in your metric that you want to detect with statistical confidence.
Because Optimizely reports and tests relative improvement, the MDE is also defined in terms of relative improvement rather than absolute improvement. Thus, use relative improvement, not absolute improvement.
For example, detecting a 10% increase on a 5% baseline means you should enter 10%.
Statistical Significance Level
Enter the Statistical Significance Level. Ensure you use the same Statistical Significance Level you set in step two in the Configure your A/B test rule section.
Number of Variations
Enter the Number of Variations, including the baseline variation.
The Visitors needed per variation updates automatically. Click Copy value to clipboard.
Complete the Frequentist (Fixed horizon) configuration
- Paste the Visitors needed per variation from the Use the Sample Size Calculator section in the Sample Size per Variation field.
- (Optional) Click + Minimum duration. Enter a whole number of days the experiment should run.
Note
- The Minimum duration must be at least one day.
- The Minimum duration is optional, but it is available if you believe that your experiment will reach its sample size before completing a full business cycle. See Seasonality and traffic spikes for information on why you should run an experiment at least one business cycle.
- Click Clear minimum duration to remove the field.
Complete your A/B test rule configuration
- Configure the remainder of your A/B test rule.
- Click Save.
- Start your A/B test rule and run your flag to launch your experiment. See Manage rulesManage rules.
You cannot pause a Frequentist (Fixed Horizon) A/B test, but you can stop it by updating the status to Conclude. If you try to conclude the A/B test rule before the sample size and minimum duration (if set) are complete, a warning displays, but you can still conclude the rule. See Conclude rule.
You cannot edit the Stats Configuration (including engine type, sample size, and duration) after starting an A/B test in Feature Experimentation. You must duplicate and start a new A/B test rule to update these settings.
Wait for completion
When you run an A/B test rule using Frequentist (Fixed Horizon) as your Stats Configuration, Optimizely checks at scheduled intervals to determine if the required sample size or minimum duration (if set) is met. Because these checks happen on a schedule, the actual number of samples collected per variation may be higher than the sample size you set in the A/B test configuration.
This extra data collection is expected behavior and is statistically valid. It does not affect the integrity of your results. Optimizely calculates and displays results based on the actual number of samples collected. This approach is standard practice in Frequentist (Fixed Horizon) testing, especially in high-traffic environments.
Optimizely does not display the Frequentist (Fixed Horizon) A/B test's confidence intervals and statistical significance until the required sample size or minimum duration (if set) is met. Instead, the results page displays the metric performance per variation during the test.
After the sample size and duration requirements (if set) are met, the Frequentist (Fixed Horizon) A/B test stops collecting data. The results page is updated and displays the same as the Experiment Scorecard in Optimizely Analytics. See Experiment Scorecard overviewExperiment Scorecard overview. You can then conclude the test, see Conclude rule.
Next steps
- To compare Frquentist (Fixed Horizon) and Sequential (Stats Engine) testing, see the Statistical analysis methods overview.
- To learn more about the statistical concepts, see Frequentist (Fixed Horizon) statistics.