New A/B Results page Summary tab
The Summary tab is where you get a quick read of your experiment and make a decision. It gives you an overview of how your experiment variations perform against your selected experiment metrics.
The summary tab has three conceptual parts:
Experiment details
The Experiment details panel shows the key information about your experiment:
- Name – The experiment name.
- Type – The experiment type, for example A/B Test.
- Status – The current experiment state, for example Running, Paused, or Concluded.
- Visitors – The total visitors in your experiment. See Target visitors with audience conditions.
- Duration – The start and end dates and the total number of full days. Optimizely truncates fractional days, so 17.8 days displays as 17.
- Project – The project where the experiment was created.
- Environment – The environment within a feature flag (Feature Experimentation only).
- Traffic allocation – The percentage of traffic assigned to each variation.
- Audiences – The audiences the experiment targets.
When an experiment is concluded, the panel also includes conclusion details, the results outcome, and the deployed variation.
Click Experiment details to open the experiment setup.
Click Hide Editor to collapse both the Experiment details and Manage metrics sections to free up horizontal space so you can review experiment results. Click Expand Editor to display it again.
Advanced options
Click Advanced options to expand the block, which lets you adjust how Optimizely calculates the results:
- Baseline – Choose which variation Optimizely uses as the comparison reference. All results display relative to this variation.
- CUPED – Turn on CUPED (Controlled-experiment Using Pre-Experiment Data), a variance reduction technique that uses historical data to help you reach statistical significance faster. See CUPED.
- Stats engine – Shows the stats engine approach you selected during experiment setup. See Statistical analysis methods overview.
- Statistical significance threshold – The confidence level results must meet or exceed to count as statistically significant. You set this during experiment setup. See Statistical significance.
Results toolbar
A results toolbar lets you refresh the data, check data quality, and adjust the data view.
- Results last updated – Shows when Optimizely last calculated the results, in your local time zone. Click Refresh results next to the timestamp forces the results page to check for the latest incoming decision and conversion events.
- SRM status – Checks for Sample Ratio Mismatch (SRM), which flags when the actual traffic split across variations differs significantly from the expected split. See Optimizely's automatic sample ratio mismatch detection.
- Check Data Health – Runs data health checks for the experiment, such as dataset uniqueness, actor alignment, and variation conflicts. See Health check overview.
Summary table
The summary table gives a high-level view of how each variation performs across all metrics, with one row per variation. It includes these columns:
- Variations – Each variation you are testing, including the original.
- Visitors – The visitor count and the percentage share of experiment traffic for each variation.
- Metric performance – The metric value and the improvement compared to the baseline.
Click Show Visitors Over Time to expand a graph of how visitor traffic split across the variations over time.
Metric performance breakdown
Optimizely splits metrics into two types: decision-making metrics and guardrail metrics. Decision-making metrics are the metrics you want to improve in your experiment. Guardrail metrics are safeguards you monitor during an experiment. They confirm that a change improving your primary metric does not harm other important metrics.
Each metric performance view provides a table with metric data and statistical calculations and over-time graphs. The primary metric carries a Primary tag. Click Expand all or Collapse all to open or close every metric section at once.
Each metric section shows a table with the following columns:
- Variations – Variations configured during experiment setup.
- Metric Value – The raw computed metric value for the variation. For example, a unique conversion metric divides the visitors who triggered the event by all visitors who saw the variation. The numerator and denominator are the raw values behind this result.
- Aggregated data columns – The columns show the aggregated data for the numerator and denominator of the selected metric type, such as unique conversions, total revenue, total conversions, and total value.
- Improvement – The relative change against the baseline.
- Confidence Interval – Shows the minimum and maximum values of the confidence interval.
- Stat Sig Level – The statistical significance level the variation reached for the metric.
Over time graphs
Over time graphs lets you review metric data more granularly. Each metric has three over time graphs that are connected to the data in each metric table: improvement over time, metric value over time, and statistical significance over time.
Statistical significance over time graph is not a representation of the historical stat sig data but rather a breakdown of stat sig calculation considering the full experiment data available at the current moment.
Manage metrics
The metrics you define during experiment setup display in the Experiment details panel, grouped as Decision-making metrics and Guardrail metrics.
You can add, remove, or change a metric directly on the results page.
Changes you make to metrics on the new results page do not sync back to the experiment setup.
The New A/B Results page calculates ratio metrics independently, without accounting for any dependency between the numerator and denominator events. This differs from the classic results page, so account for this difference when you interpret ratio metric results.
To calculate the ratio of dependent events, either add a funnel metric on the results page or run a behavioral segmentation with cohorts on the Explore tab.
Each metric includes a definition (which differs from the experiment setup) and an Advanced Settings panel. See Metrics module in Optimizely Analytics and Overview of metrics in Experimentation.
Conversion metrics use the Conversion Rate measure type. Numeric metrics use Aggregate by Property (for example, sum of revenue).
The settings in the Advanced Settings panel depend on the metric type:
- Conversion metrics (Conversion Rate) have Conversion window and Set alerts (guardrail alerts).
- Numeric metrics (revenue or value) have Conversion window and Set alerts, plus CUPED duration time interval and Outlier management.
Conversion window and alerts apply to every metric type. CUPED duration time interval and outlier management apply to numeric metrics only. The following settings are available.
Conversion window
A conversion window sets how long Optimizely counts conversions after assigning a user to a variation (also called bucketing). By default, Optimizely attributes every conversion that follows the decision event to that variation. This holds regardless of the delay, for as long as the experiment runs.
For conversion events, the reference date is the current date for a running experiment and the end date for a concluded experiment.
A conversion time window is useful for metrics where conversion typically takes some time to occur after a user is exposed to an experiment.
For example, on a bidding platform, a user may be bucketed into an experiment, but it could take several days before they actually complete a purchase. If an experiment concludes before that conversion happens, Optimizely would not capture the eventual conversion, even though the user was part of the experiment.
The conversion window addresses this by extending the period during which conversions are counted for a given user, beyond the point at which they were bucketed, ensuring delayed conversions are still attributed correctly.
CUPED duration time interval
If you toggle CUPED on, you can change the CUPED duration time interval which sets the historical data period that CUPED uses to reduce variance. By default, CUPED uses two weeks of historical data. To use a custom period, click CUPED duration time interval and adjust the duration. See CUPED.
Outlier management
Outlier management caps extreme values in a numeric metric so they do not inflate variance or distort your results. You can cap by percentile (Winsorization) or at a constant value (metric capping), at the user or event level. For the methods, steps, and examples, see Outlier management.
Guardrail alerts
Guardrail alerts let you configure metric alerts that protect business-important metrics during an experiment. Optimizely notifies you by email or Slack when a variation breaches a threshold you set, so you can act without returning to the results page. To set thresholds and configure email or Slack delivery, see Guardrail alerts.