Build an Experiment Scorecard

2 min

The Experiment Scorecard offers a streamlined approach to comparing experiment variations concerning business metrics stored in your data warehouse. It makes linking experiments to business outcomes easier and delivers deeper insights into user behavior and experiment performance.

For example, you can create algorithms for personalized content in the Flix streaming app to boost user engagement and increase revenue. The following example uses the Flix app and its demo datasets, which you can also see inside the app.

Go to + > Experiment Scorecard.

Select the metric type using the drop-down list, and then select Summary.

Choose the preferred Optimizely Feature Experimentation or Web Experimentation Experiment, and then complete the following:

Dataset – Select Users for the actor dataset that includes the users in the experiment.

Threshold – Set the statistical significance threshold to 95%.

Variations – Select the variations, which are alternate versions of a site that you test against the original in Optimizely Experimentation.

Configure the Decision-making Metrics. In this section, you can create a metric or use a previously created one.

To create a metric, you have three options: Conversion, Numeric aggregation, and Ratio. Select one from the catalog or add a new block if you want to use an already created metric. Learn more about metrics in Experiment Scorecard. For example, if you want to configure a Conversion metric, choose Conversion Rate as the measure type and Play Content as the event.

Select the metric's format. By default, it is set to Percentage.

(Optional) Add Guardrail Metrics, which are critical business indicators you closely monitor during experiments. You can choose from three options: Conversion, Numeric aggregation, and Ratio, or you can select an existing Metric. For example, select an existing metric, like Total Ad Revenue.

Choose a Baseline from the drop-down list. The option you select under Baseline is the variant against which other variants are compared. The time interval is automatically configured to match the experiment's duration using metadata from Optimizely. In this example, it is set to Since Sat, February 12, 2022.

Click Run. The scorecard displays a statistical significance table, showing which variation performed better based on the selected metrics.

Enter a descriptive name for the analysis and save it.