Contextual Bandit Results page
- Optimizely Web Experimentation
- Optimizely Personalization
- Optimizely Feature Experimentation
The Contextual Bandit Results page lets you track your contextual bandits' performance, view different experiences and audiences to examine visitor behavior, and take action. Contextual bandits score variations based on the impact on the primary metric and user attributes to serve the most personalized variation to every user.
Contextual bandits use decision-level scoping for event attribution. This means Optimizely assigns events to the decision they happened in. In other words, Optimizely associates events with the decision event that occurred before the event. Optimizely uses decision-level scoping because while the contextual bandit is running, user context can change, and the contextual bandit would optimize for this user, using their new context. Optimizely uses decision-level scoping to capture these details. See the Decision-level scope section in the How Optimizely Experimentation counts conversions documentation for information.
Optimizely calculates most metrics using a session scope within a Personalization campaign. For contextual bandits, Optimizely automatically changes this into decision scope, as it is the best way to capture their performance. Other metrics (for example, revenue per conversion and value per conversion) are still calculated per conversion, even though the UI says decisions. You should not use these metrics for contextual bandits.
View results for contextual bandits
You can view your contextual bandit results from the Campaign Results page.
To access the Campaign Results page, complete one of the following:
- From the Optimizations dashboard, click Results.
- From the Manage Campaign dashboard, click Experiences > View Results.
Then, in the Experiences section, click on a contextual bandit to open its results page.
Results are in local time, according to the time zone set on your machine.
About the Contextual bandits results page
The contextual bandit results page includes the following sections:
- Segmentation drop-down lists – Select different options to see how different segments of users behave on different dates.
- Download, Share, Edit – Download results as a PDF, copy a shareable URL, or edit the experiences.
- Summary – Displays the total number of visitors and sessions for the primary metric. A higher number of sessions than visitors suggests the contextual bandit is helpful.
- Distribution – Shows how the contextual bandit split traffic between users who saw randomized variations and those who received personalized ones.
- Top User Attributes – Lists user attributes that the contextual bandit model considered the most important when assigning users to variations. Put another way, the "weight" of the attribute when deciding to put a user into a specific variation. These attributes influenced the model's decisions the most.
- Download Results – Download the full list of user attributes provided to the contextual bandit model with their weights (relative importance).
- Metrics – Displays how each variation performed for the selected metric, including Unique Conversions, Number of Sessions, Conversion Rate, and the Relative Improvement.
- View or Hide Graph – Expand or collapse the corresponding metric's graph.
- Graph type selector – Use the drop-down list to switch between available graphs, including Decisions Over Time, Unique Conversions Over Time, and Unique Conversions per Decision Over Time.
- Graph details – Hover over the graph to inspect how each variation performs.
- Additional metrics – View performance for the remaining metrics in your contextual bandit.
The improvement percentage of the campaign summary is calculated using session scope attribution, while the improvement percentage of each CMAB experience is calculated using decision scope attribution. See How Optimizely Experimentation counts conversions.
Attribute values per variation
Click View Contributing Attributes in the metrics section to see the attributes and values associated with that variation. The expanded view shows a sub-table with two columns:
- Contributing Attribute – The attribute name (for example, Device).
- Attribute Value – The value within that attribute that was associated with conversions for the variation (for example, desktop).
For example, a variation named Variation #3 might show the following contributing attributes and attribute values:
- Ad Campaign – utm_campaign
- Location – city: NY, continent: NA, country: US, region: MO, dma: 158
- Traffic Source – direct
You can read this table as the segment that responded best to Variation #3. Visitors who arrived directly (not through a referrer), came in on the utm_campaign ad campaign, and were located in the listed region converted at a higher rate on this variation than on the others. In the screenshot, Variation #3 converted at 7.93% on Add to Cart, compared to 6.93% for the holdback, a relative improvement of +14.38%.
Each variation shows its own attributes and values, so two variations in the same experience can list different results.
Metrics reflect the primary metric
The summary, the metrics columns, and the over-time graph all reflect the primary metric configured on the experience. The metric names on the page change with the primary metric you select.
When the primary metric is Overall Revenue, the page shows the following:
- The summary shows Overall Revenue, Total Sessions, and Improvement.
- Metrics section shows revenue, number of sessions, revenue per session, and improvement for each variation.
- The over-time graph is named Revenue Over Time and plots cumulative revenue for each variation against the holdback group.
When the primary metric is a conversion event, the same areas show Unique Conversions, Number of Sessions, and Conversion Rate, and the over-time graphs include Decisions Over Time, Unique Conversions Over Time, and Unique Conversions per Decision Over Time.
The over-time graph plots the cumulative value of the primary metric across the campaign duration, with one line per variation and one line for the holdback group. The holdback group is the share of visitors excluded from the personalized experience, used to measure improvement. Hover any point on a line to see the cumulative value for that variation on that date.