Ratio metrics
- Optimizely Web Experimentation
- Optimizely Personalization
- Optimizely Performance Edge
- Optimizely Feature Experimentation
A ratio metric divides one event by another to measure the relationship between them. Use one to track performance the way your business already measures it, such as revenue per add-to-cart click and feature use per account.
Unlike simple metrics, which automatically normalize by the number of users in an experiment, ratio metrics let you define a custom denominator, such as accounts or a specific event.
You can choose an aggregation for each event: total count, unique count, total revenue, or total value. The numerator and denominator do not need to use the same aggregation.
When to use ratio metrics
Use ratio metrics in the following scenarios:
- You need to measure efficiency or value per event, not per user.
- Your key performance indicator (KPI) is already expressed as a ratio, such as the following:
- Revenue per session.
- Average order value.
- Purchases per account.
- You want to create funnel-like metrics to analyze sequential event relationships (for example, form submissions after banner clicks).
Supported aggregation combinations
While the primary use case for ratio metrics is based on \frac{\text{sum}}{\text{count}}, Optimizely Experimentation also supports different configurations, such as \frac{\text{count}}{\text{count}}. For example, create a ratio metric that tracks completed purchases per add-to-cart click, using the following structure:
\frac{\text{Total completed purchases}}{\text{Total add-to-cart clicks}}Calculation methodology
To calculate ratio metrics, Optimizely does the following:
- Check the denominator event – Optimizely checks whether the unit completed the denominator event. A unit is the user or account the metric measures.
- Check the numerator events – If the denominator event occurred, Optimizely checks for numerator events within a 48-hour window after the denominator event. The numerator events must occur within 48 hours after the denominator event to be included in the ratio metric calculation.
Note
The 48-hour window is not configurable.
- Aggregate values for the numerator events
- If using a value-based event (such as revenue), Optimizely sums the values of all qualifying events in the window.
- If using a count-based event, Optimizely counts each occurrence.
- Aggregate values for the denominator event – Optimizely applies the same rule to the denominator. It sums the values of a value-based event, or counts the occurrences of a count-based event. A unique conversion counts once per unit.
- Calculate the ratio – In each variation, Optimizely adds up the numerator values across all units (such as users or accounts) that qualified for the denominator event, adds up the denominator values across those same units, then divides the numerator total by the denominator total.
Examples
Ecommerce example
For this example, you have an ecommerce website and want to track revenue per add-to-cart click on a product details page. In this scenario, you have the following two events to track:
- Purchase completion – Total revenue.
- Add to cart clicked – Total conversions.
This example divides total revenue from completed purchases by total add-to-cart clicks.
Optimizely computes this metric by completing the following:
- Checks if the user clicks add to cart (the denominator event).
- If the user qualifies, Optimizely sums the revenue from completed purchases (the numerator event) within 48 hours.
- Divides the total revenue summed across all qualifying users by the total number of add-to-cart clicks counted across those same users, in each variation.
Marketing funnel example
For this example, you work at a company and want to track your marketing funnel by tracking form submissions after a user clicks on a promotional banner. In this scenario, you have the following two events to track:
- Form submissions – Total conversions.
- Promotional banner clicks – Total conversions.
This example divides total form submissions by total promotional banner clicks.
Optimizely computes this metric by completing the following:
- Identifies users who clicked the promotional banner (the denominator event). Optimizely includes only users who triggered this event.
- For each user who clicked the banner, Optimizely counts all form submission events (the numerator event) that occurred within 48 hours of the banner click.
- Optimizely divides the total number of form submissions counted across all qualifying users by the total number of promotional banner clicks counted across those same users, in each variation.
Streaming service example
In this example, you work for a streaming service, and you want to capture the household-level engagement for subscription-based platforms. In this scenario, you have the following two events to track:
- Total watch time (in minutes) – Total value.
- Number of accounts – Unique conversions.
This example divides total watch time by unique conversions of unique accounts.
Optimizely computes this metric by completing the following:
- Checks whether each account started streaming content (the denominator event). This event is treated as a unique conversion, meaning each account is counted once when it first starts streaming during the experiment.
- For each qualifying account, Optimizely sums the total number of minutes watched across all associated users (the numerator event). Only watch time within a 48-hour window after the streaming start event is included.
- Divides the total minutes watched across all qualifying accounts by the number of qualifying accounts in each variation. Because the denominator event is a unique conversion, each account contributes one to the denominator total.
Statistical methodology
When conducting experiments using ratio metrics, it is important to estimate the metric's variance to determine its statistical significance. Given that a ratio metric is a ratio of two events, Optimizely employs a first-order Taylor series approximation (often referred to as the Delta method) to approximate this variance.
Optimizely defines a ratio metric \hat{R} as follows:
\hat{R} = \frac{\sum_{i=1}^{n} y_i}{\sum_{i=1}^{n} x_i}The formula uses the following variables:
- x_i represents the observed values of the denominator event.
- y_i represents the observed values of the numerator event.
Optimizely calculates the approximate variance of \hat{R} as follows:
V(\hat{R}) \approx \frac{1}{n} \frac{1}{\mu_x^2} (R^2 \sigma_x^2 + \sigma_y^2 - 2R\sigma_{xy})The formula uses the following variables:
- n is the sample size.
- \mu_x is the mean of the denominator variable.
- \sigma_x^2 is the variance of x.
- \sigma_y^2 is the variance of y.
- \sigma_{xy} is the covariance between x and y.
This approximation helps in understanding the variability of the ratio metric, which is crucial for hypothesis testing. The presence of covariance (\sigma_{xy}) in the formula indicates that the two events in a ratio metric may not be independent. Instead, their values may be statistically dependent, meaning that changes in one event could be correlated with changes in the other. This dependence is captured in the variance calculation to ensure accurate statistical inferences. Optimizely's sequential testing methods were adjusted to account for this variance estimation, ensuring accurate and reliable test results.
Best practices
Ratio metrics are powerful, but require careful interpretation. Follow these best practices to ensure accuracy.
- Ratio metrics can be any type of metric – Ratio metrics can serve as your primary, secondary, or monitoring metric.
- Always include the component metrics as simple metrics – Add the numerator and denominator as simple metrics in your experiment. This helps you interpret the ratio correctly and helps with the following:
- If only one event changes drastically, the ratio metric can become skewed, potentially causing misleading conclusions.
- If the ratio remains the same, but the total counts of both events drop dramatically, it could indicate an underlying issue not apparent from the ratio metric alone.
- Choose the right metric type – Start with a simple metric when it adequately captures your goal. Use a ratio metric when a custom denominator is essential.