Primary metrics, secondary metrics, and monitoring goals

The right metrics can validate or disprove your hypothesis and help you progress toward your business goals. In experience optimization, metrics have different roles depending on where you set them and what you want to know.

The following are a few tips for setting goals and metrics:

In Optimizely, you set a primary metric for each experiment or campaign to determine its success. It is the most important goal of the experiment and proves or disproves your hypothesis.

Use the following questions to choose a primary metric:

Consider setting a primary metric on the same page as your change. Many optimization teams automatically track revenue per visitor as the primary metric. Top-level metrics like revenue and conversion rate are important, but these events are often far away from the changes made, and your test may take a long time to reach statistical significance or end up inconclusive. 

Stats Engine uses false discovery rate control to help you make better business decisions. The primary metric's false discovery rate control is done in a way that is independent of secondary and monitoring metrics. This means that the primary metric's evaluation is separate and independent of other metrics. See the Stats Engine approach to metrics and goals section for information.

For example, you test the design and placement of an Add-to-Cart button. Your business cares about revenue, which is measured five pages down the funnel. You may devote a lot of traffic to this test and risk an inconclusive result.

Instead, you decide to measure clicks to the Add-to-Cart on product pages. This is a primary metric directly affected by the changes you make. 

If you are testing bulk discounts on your site, your primary metric might be conversion rate or average order value (AOV). Neither metric fully accounts for the behavior you are trying to affect.

The conversion rate could rise as customers are incentivized or decrease as customers wait to create large, discounted orders. AOV could rise as customers buy more in bulk or decrease as discounts replace full-price orders.

From this perspective, revenue-per-visitor is the best metric. It equals the conversion rate (how often customers purchase) multiplied by the AOV (how much they spend). It is the best overarching goal in this test, where smaller goals may provide conflicting information.

Secondary metrics help you gather insights for long-term success. Optimizely lets you rank secondary metrics from two to five. Secondary metrics track long-distance events and more ambitious metrics, such as end-of-funnel events like order value and order confirmation. These events provide valuable information but may reach significance slower as a primary metric.

Secondary metrics also give you visibility across the different funnel steps. The significance threshold for secondary metrics is adjusted based on the number of metrics and variations to ensure proper false discovery rate control. This has the consequence that the more secondary metrics you have, the longer it may take for each to reach statistical significance. Secondary metrics do not impact the primary metric's speed to significance because the primary metric is evaluated separately. See the Stats Engine approach to metrics and goals section for information.

For example, if you change your product page and display shipping costs, your secondary metric might measure the change in drop-offs from the shipping page in your funnel. Use secondary metrics to learn when visitors drop off or return to the home page and how these patterns compare between the original and variations.

Here is a list of common secondary metrics and the reason for tracking:

Monitoring goals are goals and events that are not your primary or secondary metrics. They help you gather insights that are key to long-term success but are diagnostic. Monitoring goals track whether your experiment moves visitors in the right direction. Your change might create adverse effects in another metric. Monitoring goals form a warning system that alerts you when you hurt another revenue path.

The false discovery rate control calculations give each monitoring metric a fractional weight. Monitoring metrics are evaluated with a weight of 1/n, where n is the number of monitoring metrics. For example, if there are two monitoring metrics, each would be given a weight of 1/2. This approach ensures that monitoring metrics have minimal impact on the evaluation of secondary metrics and no impact on the primary metric. See the Stats Engine approach to metrics and goals section for information.

For example, you show visitors more products on the product category page. With your primary metric, you find that people view more products. Here are some other questions the monitoring goal can help you find out:

Here are common monitoring goals and their reason for tracking:

When you run an experiment with many variations and metrics using traditional statistics, you increase the chance of false positive results. Your test data may show a significant difference between your original and your variation, but it is just noise in the data. This makes it harder to declare winners.

Stats Engine uses false discovery rate control to reduce your chance of making an incorrect business decision or implementing a false positive among conclusive results.

Stats Engine works the same for revenue-per-visitor goals as for other goals. You can look at your results and get an accurate assessment of your error rates on winners and losers and confidence intervals on the average revenue per visitor (RPV).

However, when interpreting your results for RPV goals, you should be aware of some differences. Testing for a difference in average revenue between a variation and baseline is more challenging because revenue distributions are often heavily skewed. This impedes the distributional results for many techniques, including t-tests and Stats Engine. They have less power to detect differences in average revenue when those differences exist.

Optimizely's Stats Engine regains some of this lost power through skew correction. Skew corrections were designed to work well with other aspects of Stats Engine. The underlying skewness of the distributions is correctly factored into the shape of the confidence interval. Detecting differences in average revenue is more reasonable for the visitor counts that Optimizely regularly sees in A/B tests.

In a funnel, the most immediate effects are directly downstream from your changes. The measurable impact is bigger when an event is closer to the change. As you move downstream, the signal lessens as visitors from different paths and motivations enter the stream. The effect may not be measurable at the end of the funnel.

Metrics with lower conversion rates require more visitors to reach statistical significance. Events further from the page have a lower improvement in conversion rates as visitors enter from different paths, leave the site before they convert, and more. Your test will take longer to reach significance.

Consider setting a primary metric on the same page as your change. This finds the impact of your change immediately, so you can find a winning variation faster. Quick wins help generate credibility and interest in your testing program and provide fast, reliable insights about visitor behavior. You can build a data-rich testing program to iterate on its insights.

Revenue and conversion rate make excellent secondary metrics and help keep your program focused on long-term success.