Experiment readouts
Use an experiment readout to share an experiment's results with stakeholders on one page. A readout is a dashboard with a verdict, the scorecard, and a chart for each metric. Stakeholders see the outcome without interpreting the scorecard. You also skip writing a results summary by hand.
A readout is a snapshot. Analytics calculates the verdict and the scorecard figures when you generate the readout. Later scorecard edits and incoming data do not change the readout until you regenerate it.
This page covers how to generate, read, and regenerate a readout. To configure the scorecard that a readout uses, see Experiment analysis.
Prerequisites
A readout uses the saved experiment scorecard. Before you generate a readout, add at least one metric to the scorecard and save the experiment. Save or discard any unsaved changes first. Generate Readout stays unavailable until you save a scorecard with a metric.
Generate a readout
Generate a readout to report results, for example at a review meeting or at the end of an experiment.
To generate a readout, complete the following steps:
- Go to Experiments.
- Click an experiment.
- Click the Readout tab.
- Click Generate Readout.
When generation finishes, the Readout tab displays the readout dashboard.
Read the readout
The readout gives you the experiment outcome first, then the evidence behind it. A header line shows when Analytics generated the readout and the last date of data it includes.
The readout dashboard contains the following tiles:
- Summary – Shows the experiment name, the date range of the readout, and the verdict. For each decision-making metric, the tile then lists the leading variation, its improvement, and its confidence interval. The tile also shows the significance and sample size of each lead.
- Scorecard – Shows the experiment scorecard table with the figures from when you generated the readout.
- Metric charts – Shows one time-series chart for each metric on the scorecard.
The readout covers the experiment from its start date to its end date. For a running experiment, the range ends when you generate the readout.
Understand the verdict
The verdict tells you in one sentence whether the experiment has a winner. The same results produce the same verdict every time.
The verdict considers the metrics in the Decision-making metrics section of the scorecard. It compares each variation against the baseline, which is the variation that you compare the others against. When the scorecard has no decision-making metrics, the verdict considers every metric. Each metric's direction setting decides whether an increase or a decrease counts as an improvement.
The verdict takes one of the following forms:
- Winner – One variation beats the baseline on every decision-making metric, and each lead is statistically significant. The verdict names the variation and its improvement on each metric.
- Ahead but not yet significant – One variation leads on every decision-making metric, but some leads are not yet significant. The verdict names the metrics that have not reached your statistical significance threshold.
- No clear winner – The decision-making metrics favor different variations. The verdict also reports no clear winner when a metric has no variation that beats the baseline.
- Insufficient data – The decision-making metrics have no results yet.
The verdict also names each variation that performs significantly worse than the baseline on a decision-making metric. This statement displays with every form of the verdict, so a significant loss never goes unreported.
Add a narrative with Mark
When you generate a readout, Mark adds a written narrative to it. The narrative explains the outcome in plain language and recommends next steps. Your readout then tells the story of the experiment, not only its numbers.
Narratives have the following prerequisites:
- You must use Opti ID to access Mark.
- You must have generative AI enabled in Optimizely.
When a readout finishes generating, Mark chat opens and asks Mark to add a narrative. Mark then completes the following tasks:
- Reads the experiment scorecard results.
- Builds funnel, paths, or segmentation explorations from your events to compare user behavior between variations.
- Adds each exploration that shows a clear difference between variations to the readout. Mark leaves out explorations without a clear difference.
- Adds a narrative text tile to the readout.
- Tells you in Mark chat what it added to the readout.
The narrative restates the verdict word for word and describes what each added exploration shows. The narrative then recommends next steps that match the verdict. Mark does not recommend shipping a variation that has not won. The narrative also flags any guardrail metric that is significantly worse for the winning variation. A guardrail metric tracks a result that must not get worse. Review these flags before you ship the winner. The narrative uses only figures from the verdict and the scorecard results.
Mark changes the readout only once for each generation. To ask more about the results, continue the conversation in Mark chat. Mark answers in chat and leaves the readout unchanged.
Mark does not add a narrative in the following cases:
- You view the experiment results embedded in Web Experimentation or Feature Experimentation.
- The readout has no verdict, for example because the scorecard returned no results.
Regenerate a readout
Regenerate a readout to report up-to-date results, for example after the experiment collects more data. Also regenerate it after you change the scorecard. A regenerated readout replaces the current one, and the previous version is no longer available.
To regenerate a readout, complete the following steps:
- On the Readout tab, click Regenerate Readout.
- In the Regenerate this readout? window, click Regenerate.
Analytics recalculates the verdict and figures from the saved scorecard, and the header shows the updated generation time. When narratives are available, Mark adds a narrative to the regenerated readout.
Troubleshoot readout generation
Use these fixes to get a readout when generation does not finish. The Readout tab displays one of the following messages:
- Timeout message – Generation took longer than expected. The readout sometimes finishes on the server without reaching your browser. Click Generate Readout or Regenerate Readout again. A retry replaces any readout that the earlier request created.
- Error message – Generation failed for another reason. Click Generate Readout or Regenerate Readout to try again.