CUPED (Controlled-experiment Using Pre-Experiment Data)
CUPED (Controlled-experiment Using Pre-Experiment Data) is a statistical method that reduces variance in A/B tests, enhancing their sensitivity and making it easier to detect differences between groups. By lowering variance, CUPED lets experiments achieve statistical significance with less data if there is a true treatment effect. CUPED helps address common experimentation challenges such as insufficient traffic and high variance, so you can run more efficient A/B/n tests.
Consider the following when using CUPED:
- Metric compatibility – This feature is available only for numeric metrics: it is mathematically inapplicable to conversion metrics.
- Covariate selection – A covariate is the pre-experiment data CUPED uses to adjust your results. By default, CUPED uses the pre-experiment values of each metric itself as the covariate, because that typically has the highest impact. Select any other metric, or any Boolean or numeric visitor property, as a CUPED variable instead. Choose a covariate that correlates with the metric under test. A covariate unrelated to that metric produces no variance reduction.
- Warehouse support – Warehouse-Native Experimentation Analytics supports CUPED on SnowflakeSnowflake, BigQueryBigQuery, and DatabricksDatabricks. Contact Optimizely with requests for additional warehouse support. The A/B Results page runs CUPED on data Optimizely hosts, and needs no warehouse connection.
- Health checks – CUPED has no health check for covariate imbalance, similar to a sample ratio mismatch (SRM) check. Covariates that are unbalanced across variations sometimes increase variance, especially when pre-experiment data is sparse.
- Pre-experiment data requirement – By default, CUPED reads data from two weeks before the experiment start date. The window ends at each visitor's first decision event. CUPED reduces variance only when that window holds history for the covariates in use. A metric created for this experiment has no history, so CUPED has no effect on it. This applies to both Optimizely-hosted data and warehouse data. When an experiment measures only newly created metrics, select a CUPED variable that does have history. An established metric, or a numeric or Boolean visitor property, works.
CUPED runs in two places and behaves the same way in both:
- A/B Results page – Uses the data Optimizely collects and hosts. Requires no data warehouse connection.
- Warehouse-Native Experimentation Analytics – Uses decision and event data in your own data warehouse.
Enable CUPED
- Create an Experiment Scorecard or open the New A/B Results page.
- Expand Advanced options.
- Choose the preferred Experiment using the selector.
- Enable the CUPED toggle.
- Hover over the toggle and click Settings to open CUPED settings. In CUPED settings, you can control the variables CUPED uses. Click Add variable to add another, then click Save.
- Click Run to update the visualization.
- Click Save to save the experiment with CUPED enabled.
By default, CUPED applies only to the first three decision-making metrics in the experiment. To apply CUPED to any other metric, turn it on in the Advanced section of that metric.
If you toggle CUPED on for a numeric metric, you can change the CUPED duration time interval which sets the historical data period that CUPED uses to reduce variance. By default, CUPED uses two weeks of historical data. Click Reset to revert to the default duration.
In the Analytics Experiment Scorecard template, using CUPED impacts your experiment metrics' variance reduction and sensitivity.
The following Experiment Scorecard shows the same experiment with CUPED turned off and turned on. Compare the confidence interval on each metric.
With CUPED
Without CUPED
Statistical methodology
Optimizely CUPED model is a regression-based covariance adjustment method, which is more advanced than the standard CUPED method. It uses a linear regression model to filter out noise by utilizing pre-experiment data:
Y_i = \alpha + \beta T_i + \vec{\gamma}^{\,T} \tilde{X}_i + e_i \quad (i = 1, \ldots, n)The regression equation uses the following variables:
- Y_i – Value of a target metric (numeric metric) for person i.
- T_i – Treatment indicator (0 for control, 1 for treatment).
- \tilde{X}_i – Centered covariates derived from the pre-experiment data.
- \alpha, \beta, and \vec{\gamma} – Regression coefficients.
- e_i – Error term.
The ordinary least squares (OLS) estimator \hat{\beta} is an unbiased estimator for the true treatment effect.
Having pre-experiment data \tilde{X}_i in the model reduces the variance of \hat{\beta} (that is, variance reduction), resulting in a higher chance of seeing a significant result.
When \tilde{X}_i contains only the historical value of Y_i, the Optimizely method is identical to standard CUPED. Additional covariates make the Optimizely method more general than standard CUPED.
In practice, the Optimizely method works in two steps. The Frisch-Waugh-Lovell theorem shows that these steps produce the same treatment effect estimate as the full regression:
- Regress Y_i on \tilde{X}_i to get the predicted values \hat{Y}_i.
- Compute the residuals Y_i - \hat{Y}_i and feed the residuals into the Stats Engine.
When CUPED does not reduce variance
CUPED reduces variance only when the pre-experiment data predicts the metric under test. Expect little or no change in the following cases:
- CUPED is off for the metric. By default, CUPED applies only to the first three decision-making metrics.
- No pre-experiment history exists for the covariates in use, such as a metric created for this experiment.
- The covariates relate only weakly to the metric under test.
- Pre-experiment data is sparse, which leaves covariate values unbalanced across variations.
Results remain valid in these cases. They match the results the experiment produces without CUPED. Turning CUPED on does not reduce the significance of a metric. If significance drops after you turn on CUPED, contact Optimizely Support with a link to the results page.