---
title: Configure a Frequentist (Fixed Horizon) A/B test
slug: experimentation-strategy/docs/configure-a-frequentist-fixed-horizon-a-b-test
createdAt: 2026-08-05T20:33:25.528Z
---

This article applies to the following products:

- Optimizely Feature Experimentation
- Optimizely Analytics

Configure a Frequentist (Fixed Horizon) A/B test to evaluate results one time, at a sample size you set in advance. This approach suits teams that need a traditional A/B test with a strict analysis plan.

## Prerequisites

You need a flag in Feature Experimentation before you begin. See [Manage flags](docId\:oxWAaDQgCSG0Z2Zw6jZn2).

## Create a Frequentist (Fixed Horizon) A/B test rule

Create an A/B test rule that uses the Frequentist (Fixed Horizon) method. The rule then evaluates results against a fixed sample size.

### Create an A/B test rule

Start with a standard A/B test rule on your flag:

1. Select a flag from your Feature Experimentation project.
2. Create an A/B test rule in the environment you want. See [Run A/B tests in Feature Experimentation](docId\:oqDGF6JrR-nvfIShOH8c8).

### Configure your A/B test rule

Set the statistical method and significance level that the test uses to decide its result:

1. Configure your rule as you would any A/B test rule.
2. Complete the following for **Stats Configuration**:
   1. Select **Frequentist (Fixed Horizon)** for **Stats Configuration**. This setting replaces the default Sequential (Stats Engine) method with the Frequentist (Fixed Horizon) method. To compare the two methods, see [Statistical analysis methods overview](docId\:sF8u1ST65p2LijPMqQ5eQ).

      ![Screenshot of the Stats Configuration drop-down list that shows the Frequentist (Fixed horizon) option](https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/YSvyWgsgr9XJvwn2VB4q1-20260720-144803.png)
   2. (Optional) Update **Statistical Significance Level**. The level defaults to the value in the project **Settings**. See [Change the statistical significance setting in Optimizely Experimentation](docId\:LnxmwNc8bywfq5MG0P_3Z).
      Optimizely recommends a level of 90% or 95%. The level must be between 70% and 99%.

      :::hint{type="info"}
      **Note**

      Use the same significance level here and in the [Frequentist (Fixed Horizon) Sample Size Calculator](docId\:gPNnVD_zJq8OOgMLaZtau) in the next section.
      :::

## Use the Frequentist (Fixed Horizon) Sample Size Calculator

Use the Sample Size Calculator to find how many visitors each variation needs before you analyze results. The sample size is a lifetime total for each variation. Optimizely does not reset it daily or weekly. See [How visitors count toward the sample size](docId\:L08Xi7fKTSYjVxzNAXY3I).

The calculator assumes a statistical power of 80% and an even traffic split between variations. Click **Sample Size Calculator** and enter the following information:

- [Metric Type](docId\:gPNnVD_zJq8OOgMLaZtau)
- [Baseline Metric Value](docId\:gPNnVD_zJq8OOgMLaZtau)
- [(Optional) Metric Variance](docId\:gPNnVD_zJq8OOgMLaZtau)
- [Minimum Detectable Effect](docId\:gPNnVD_zJq8OOgMLaZtau)
- [Statistical Significance Level](docId\:gPNnVD_zJq8OOgMLaZtau)
- [Number of Variations](docId\:gPNnVD_zJq8OOgMLaZtau)

### Metric type

Select the type of metric you want to analyze from the **Metric Type** drop-down list. The type determines whether the calculator also needs a variance value.

- **Conversion** – Binary metrics that track whether an event happened, such as a visitor converting or not. Examples include sign-ups or completed purchases.
- **Numeric** – Metrics that measure continuous values, such as revenue, items added to a cart, or total clicks per visitor.

### Baseline metric value

Enter the current value of the metric you are testing, before any experiment changes. The calculator measures the minimum detectable effect relative to this baseline.

For example, if your sign-up rate is 5%, enter *5*.

### Metric variance (optional)

If you selected **Numeric** for **Metric Type**, enter **Metric Variance**. Conversion metrics do not need a variance value.

Variance measures how spread out the metric values are. Higher variance means the test needs more visitors to detect the same effect.

To calculate variance yourself, use the following formula:

```tex
\text{sample variance} = \frac{\sum_{i=1}^{n}(x_i - \bar{x})^2}{n - 1}
```

For example, suppose your metric is minutes spent in the app per user. Five users have values of 4, 6, 5, 9, and 6. Calculate the variance with the following steps:

1. Find the average of the values: (4 + 6 + 5 + 9 + 6) ÷ 5 = 6.
2. Subtract the average from each value. The results are −2, 0, −1, 3, and 0.
3. Square each difference. The results are 4, 0, 1, 9, and 0.
4. Add the squared differences: 4 + 0 + 1 + 9 + 0 = 14.
5. Divide the total by *n* − 1, where *n* is the number of values: 14 ÷ (5 − 1) = 3.5.

The variance is 3.5.

### Minimum detectable effect (MDE)

Enter **Minimum Detectable Effect**. The minimum detectable effect (MDE) is the smallest change in your metric that you want to detect with statistical confidence. A smaller MDE detects subtler changes but requires more visitors.

Enter the MDE as a relative improvement, not an absolute one, because Optimizely reports and tests relative improvement. For example, to detect an increase from a 5% baseline to 5.5%, enter *10%*.

See [Use minimum detectable effect when designing an experiment](docId:0NHL0ppkIJjpuipng99bL).

### Statistical significance level

Enter **Statistical Significance Level**. Use the same level that you set in step 2 of [Configure your A/B test rule](docId\:gPNnVD_zJq8OOgMLaZtau). A mismatch makes the sample size inconsistent with how the test evaluates results.

### Number of variations

Enter **Number of Variations**, including the baseline. For example, a test with one baseline and one treatment has two variations.

![Screenshot of the Frequentist (Fixed horizon) Sample Size Calculator where each input field and the Visitors needed per variation result display](https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/t4FPM3PgB3zhK810pEkQ6-20260720-144803.png)

The calculator updates **Visitors needed per variation** as you enter values. Click **Copy value to clipboard**.

## Complete the Frequentist (Fixed Horizon) configuration

Enter the sample size and an optional minimum duration so that Optimizely knows when the test has collected enough data:

1. Paste the **Visitors needed per variation** value from [Use the Frequentist (Fixed Horizon) Sample Size Calculator](docId\:gPNnVD_zJq8OOgMLaZtau) into the **Sample Size per Variation** field.
2. (Optional) Click **+ Minimum duration**.
3. (Optional) Enter the minimum number of days the experiment must run, as a whole number.

   :::hint{type="info"}
   **Note**

   - The **Minimum duration** must be at least one day.
   - Set a minimum duration when you expect the experiment to reach its sample size before it completes a full business cycle. To learn why an experiment should run for at least one business cycle, see [Seasonality and traffic spikes](docId\:ikUjNznFN30WU40MNnN6W).
   - To remove the field, click **Clear minimum duration**.
   :::

   ![Screenshot of the Stats Configuration settings that shows Sample Size per Variation and a seven-day Minimum Duration](https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/aX9k4PnDoPNSkm9S01G-d-20260720-144803.png)

## Complete your A/B test rule configuration

Save and start the rule so that the test begins collecting data:

1. Configure the remainder of your A/B test rule.
2. Click **Save**.
3. Start your A/B test rule.
4. Run your flag to launch the experiment. See [Manage rules](docId:3z-iUZ7ujSH8nL14sSKVP).

A Frequentist (Fixed Horizon) A/B test has no pause option. To stop the test, change its status to **Conclude**. If you conclude the rule before it meets the sample size and minimum duration (if set), a warning displays. The warning does not prevent you from concluding the rule. See [Conclude rule](docId\:gPNnVD_zJq8OOgMLaZtau).

Feature Experimentation locks the **Stats Configuration** when you start the test, including the engine type, sample size, and duration. To change these settings, duplicate the rule. Then start the duplicate.

## Wait for completion

Know what the results page shows while the test runs so that you do not act on incomplete data.

When an A/B test rule uses Frequentist (Fixed Horizon) as its **Stats Configuration**, Optimizely checks progress at scheduled intervals. Each check determines whether the test reached the required sample size and minimum duration (if set). Between checks, the test keeps collecting data. As a result, each variation often collects more samples than the sample size you set.

The extra data is normal and statistically valid, and it does not compromise your results. Optimizely calculates and displays results based on the actual number of samples collected. This approach is standard practice in Frequentist (Fixed Horizon) testing, especially in high-traffic environments.

Confidence intervals and statistical significance display only after the test meets the sample size and minimum duration (if set). Until then, the results page shows the visitors and improvement for each variation. It hides the per-metric tables and graphs. A message names the condition that the test has not yet met:

- `Sample size condition has not been reached.` – The message shows the visitor count of the smallest variation, the sample size per variation, and the approximate days remaining.
- `Minimum duration condition has not been reached.` – The message shows the days elapsed and the minimum duration.

If you conclude the test early, the results page displays confidence intervals and statistical significance for the data so far. Early results carry a higher risk of false positives, so treat them with caution.

After the test meets the sample size and minimum duration (if set), the test stops collecting data. The results page then matches the Experiment Scorecard in Optimizely Analytics. See [Experiment Scorecard overview](docId:7A_-hkbjJKMcUYA4-VZ-d). Then conclude the test. See [Conclude rule](docId\:gPNnVD_zJq8OOgMLaZtau).

## Next steps

Continue with the following articles:

- To compare Frequentist (Fixed Horizon) and Sequential (Stats Engine) testing, see [Statistical analysis methods overview](docId\:sF8u1ST65p2LijPMqQ5eQ).
- To learn the statistical concepts behind Fixed Horizon testing, see [Frequentist (Fixed Horizon) statistics](docId\:L08Xi7fKTSYjVxzNAXY3I).