How Optimizely Experimentation handles outliers

In statistics, an outlier is an observation that has an abnormally higher or lower value than other observations in a data set. Outliers can severely skew the accuracy of any data set analysis and lead to potentially incorrect conclusions.

Outliers can occur during your experiment, usually resulting from unusual or unexpected behavior by a customer.

For example, suppose you are running an experiment that aims to improve the average order value for your ecommerce site. Your visitors usually submit orders with an average total value of $200. Imagine a small number of visitors who submitted orders with 100 or even 1000 times higher value than the average. If the result calculations included these extreme orders, they could introduce bias into your A/B comparison and lead you to draw the wrong conclusions from your experiment.

For this reason, Optimizely lets you use outlier smoothing on your experiment results. This feature is currently available for all revenue metrics, including Overall Revenue in A/B experiments.

When outlier smoothing is enabled, Optimizely first identifies any values that exceed the daily exclusion threshold, which is three standard deviations higher than the arithmetic mean of your metric.

arithmetic mean + (3 * standard deviation)

These extreme values are designated as outliers. It is important to note that there may be no values that fall outside the custom threshold for that day, so in other words, no outliers for that day.

Next, Optimizely replaces these outliers with the metric's harmonic mean value. This process is known as outlier smoothing.

Optimizely recalculates the daily exclusion threshold each day, using a moving average of the arithmetic mean and standard deviation of your metric over the previous seven days. This process repeats for each day of the experiment.

If your account has access to the feature, you see an Enable outlier smoothing for revenue checkbox on the Metrics page. Selecting the option ensures that outliers for the revenue metrics in that experiment are automatically detected, and the harmonic mean of the metric replaces their values.