Optimizely Experiment Results page

A new A/B results page is available with CUPED, configurable stats engines, cohort segmentation, and additional data health checks. See New A/B Results page.

Use the Optimizely Experiment Results page to measure your A/B test success. Review each metric and variation to see how visitors respond to your site or application changes. Use segments to investigate visitor behavior in more detail.

Optimizely Experimentation's Stats Engine powers the Experiment Results page. It provides a data-rich view of visitor interactions, includes confidence intervals, and applies false discovery rate control.

The following details are important to know:

To access the Optimizely Results page

At a high level, the Experiment Results page displays the following:

Optimizely Experimentation's automatic SRM detection detects any early deterioration of the experiment. An SRM occurs when the traffic distribution between variations in a Stats Engine A/B test becomes severely and unexpectedly unbalanced. To rapidly detect deterioration caused by mismanaged traffic distribution, Optimizely continuously checks traffic counts throughout an experiment. If Optimizely detects a traffic imbalance, the health indicator updates immediately. If your Experiment Health displays anything other than Good, you must investigate why the traffic imbalance occurred. See Optimizely automatic sample ratio mismatch detection for information.

Optimizely Experimentation does not check for visitor imbalances for the following:

The primary metric displays first and is expanded by default. Secondary metrics are collapsed automatically. Click on a secondary metric name to open that single secondary metric, or click Expand all to display every metric.

You can toggle between different graphs for each metric. To see or hide charts, click Hide graph or View graph. Use the drop-down list to select different sets of data.

The x-axis of the graphs represents the end time of each time bucket used in the Stats Engine calculation. Although the axis displays date values, the graphs are not plotted by calendar date. Instead, they visualize 100 equally divided time buckets that span the entire duration of the experiment at the time of calculation.

Because time bucket durations change as the experiment runs, comparing the graph across different calculation times may be misleading. The graph is a snapshot of the current state.

Segmenting lets you filter your Optimizely Experiment Results page to see if certain groups of visitors behave differently from your visitors overall. See Segment your results for default Web Experimentation segments.

Use the Date Range drop-down list to select start and end dates for your results page. Click Apply. The results generated are in your computer's time zone.

By default, Optimizely Experimentation displays results for visitors who enter your experiment. However, not all visitors behave like your average visitors. By segmenting your results, you can gain deeper insights into your customers to design data-driven experiments.

Segments and filters should only be used for data exploration, not for making decisions.

Use the Segment drop-down list to learn about a segment of your visitors. You can segment by a single attribute value or a combination of attributes.

All attribute value options are displayed for segmenting, regardless of whether the visitors of a specific experiment have sent those attribute values to Optimizely Experimentation. Because attributes are shared across all Experimentation projects, the Segment drop-down list displays any attribute created in any project. It does not show attributes that have not been associated with a user in any experiment. There is a maximum of 500 values that can be shown per attribute.

For example:

Experiment 1 – Mobile-only A/B test with the audience set to mobile only. The visitors in this experiment send Optimizely Experimentation the attribute mobile_os, stating if they are using Android or Apple devices. The Segment drop-down list displays all attribute values for browser that Experiment 2 sent to Optimizely.

Experiment 2 – Desktop-only A/B test with the audience set to desktop/laptop only. The visitors in this experiment send Optimizely Experimentation the attribute browser with their browser name. The Segment drop-down list displays all attribute values for mobile_os with the options of Android and Apple.

When selecting a segment outside the intended experiment, the page displays 0 and the attribute value options.

See Segment your results on Optimizely Web Experimentation for default segments. You can also segment by up to 100 custom attributes, and 500 values can be shown per attribute in Optimizely Web Experimentation or Optimizely Feature Experimentation.

Sometimes you may want to see how your variations compare to one variation in particular, which may not be the original. Use the Baseline drop-down list to select a different variation as the baseline.

Click Share and copy the URL provided to send your Optimizely Results page to critical stakeholders.

The Share link only gives access to the A/B test's results. Users of the link can segment data, view charts, filter by date range, and more.

If you want to reset the link you shared, click Reset Link. Users with the previous link can no longer access the Optimizely Experiment Results page.

Use the Export CSV feature to download the results of your A/B test in a comma-separated value (CSV) file. You can also share the raw results with others, store the data on your machine, or perform additional analysis.

Click Export to download the CSV file of the results displayed on the page (limited to the Date Range and Segment selected).

See Reference for columns in CSV export files for campaigns and experiments. You can also access your Optimizely Experimentation export data in other ways.

Click More > Manage Metrics to add or remove metrics or set a different primary metric.

Stats Engine ensures that the primary metric (which signals whether a variation "wins" or "loses") reaches significance as quickly as possible.

Click Edit Experiment to make changes to your experiment. Use this option to pause a variation or adjust your traffic distribution.

Resetting your Optimizely Experiment Results page affects the data displayed on the page and what Stats Engine calculates for this experiment.

The raw data for your experiment is still available after the reset. If you need to access it, see Access Optimizely Experimentation Event Export data.

After you click Reset Results, a confirmation page displays an agreement to verify that you understand this is a permanent action for the Optimizely Experiment Results page.

You must select all three boxes to click Reset Results.

You may need to refresh the page to see your change reflected. The Optimizely Experiment Results page now displays as though the experiment has just started, and there is no data yet. Visitor counts, experiment duration, and metrics return to 0. A notation displays the date of the last reset, and the Date Range only shows the current date.

Clicking Reset Results on your Optimizely Experiment Results page does not reset variation bucketing assignments. Resetting the results is a front-end "wipe." As stated previously, the underlying data events for the past results are still available through data exports. When the results are reset, visitors continue to see the variation they were previously bucketed into and count toward the correct variation.

Optimizely Experimentation caches your result data for displaying on the Optimizely Experiment Results page. Optimizely Experimentation also fetches the latest data and refreshes the results page automatically.

When Optimizely Experimentation cannot fetch the cached data or the results are not yet calculated, Optimizely Experimentation calculates the data in the backend and automatically reloads the page.

In case of a fatal error, you see the following:

After a few minutes, try refreshing the page. If the problem persists, see the troubleshooting Optimizely Experimentation documentation or contact Support.