Experimentation distribution methods
- Optimizely Web Experimentation
- Optimizely Personalization
- Optimizely Feature Experimentation
- Optimizely Full Stack (Legacy)
The intent of your test determines when you should run an optimization using Stats Accelerator, multi-armed bandit (MAB), or a contextual bandit distribution for the variations' traffic allocation.
- To discover which variation improves your product with statistical certainty – Run an experiment with Stats Accelerator. See the When to use Stats Accelerator section.
Note
Stats Accelerator's statistical model is a variation of a multi-armed bandit.
- It helps you capture more value from your experiments by reducing the time to statistical significance.
- It minimizes time by monitoring ongoing experiments and using machine learning to adjust traffic distribution among variations.
- Traffic is routed to the variation most different from the baseline, regardless of whether it is better or worse until statistical significance is reached; then, traffic is routed to the other variations.
- To optimize towards a primary metric – Use a multi-armed bandit (MAB) optimization. See When to use a MAB optimization.
- It helps you maximize the performance of your primary metric across your variations by dynamically re-allocating traffic to whichever variation is currently performing best.
- It minimizes regret because traffic is routed to variations with the best performance. The variation with the best performance is the one that has made the most impact on the primary metric.
- MABs help you extract value from the leading variation during the experiment lifecycle by avoiding the opportunity cost of showing sub-optimal experiences.
- To optimize towards a primary metric and user attributes – Use a contextual banditcontextual bandit. See When to use a contextual bandit.
- It helps you deliver the most personalized variation to each user by dynamically re-allocating traffic based on the primary metric and user attributes.
- It maximizes impact because each user is served the most personalized variation depending on user context. For every new session, a user will receive the winning variation to increase conversion rates.
- Contextual bandits are ideal to be run on pages where the content is not changed too frequently, such as the homepage, as model performance improves over time.
When to use Stats Accelerator
Stats Accelerator manipulates traffic to minimize time to statistical significance. It monitors ongoing experiments and uses machine learning to adjust traffic distribution among variations.
The following examples show situations where an A/B test with Stats Accelerator is the best choice.
- Call To Action (CTA) – Optimize a page for lead completion by testing many different copy options for a call to action.
- Landing page – Test several combinations of landing page copy, concept, and design to optimize for registrations, sign-ups, and donations.
- Add-to-cart rate optimization – Increase your add-to-cart rate by showing visitors different default images on product pages or search results.
- Search results optimization – Nudge users toward specific options (such as a travel site encouraging visitors to select a particular flight) by showing different results.
- Drive traffic to specific pages – Increase clickthrough rates on recommended articles by changing the location of the Recommended Content section.
See Stats Accelerator for information.
When to use MABs
MABs maximize conversions for short, temporary experiences. MABs answer, "Which variation shows the largest reward?"
The following examples show situations where a MAB optimization is better than a traditional A/B experiment.
- Promotions and offers – Users who sell consumer goods on their site often focus on driving higher conversion rates. You can offer special promotions that run for a limited time; for those, the changes you make are not intended to be permanent, and an MAB optimization sends more traffic to the over-performing variations and less traffic to the underperforming variations for the duration of the promotion.
- Headline testing – Headlines are short-lived content that lose relevance over time. If a headline experiment takes as long to reach statistical significance as the lifespan of a headline, then any insights gained from the experiment are irrelevant in the future. A MAB optimization lets you maximize your impact without worrying about balancing experiment runtime and the lifespan of a headline.
- Webinar – You can boost registration for webinars or other events by experimenting with several different versions of your landing page.
Note
MAB optimizations do not generate statistical significance. Instead, they push traffic to the better-performing variations. See Maximize lift with multi-armed bandit optimizations for information.
When to use contextual bandits
A contextual bandit maximizes conversions by showing variations that match user attributes. Contextual bandits score variations based on impact on the primary metric and user attributes to serve the most personalized variation to every user.
You should use contextual bandits when user characteristics (such as device, location, or interests) impact what variation they would prefer to see. For example, a financial institution collects user attributes on credit cards, credit scores, loans, annual income, and spending categories. They build variations for their current offers of a travel credit card, auto loan refinancing, and a business credit card.
For a standard MAB, you would choose a primary metric, such as a call-to-action click, and display random variations to users. When one variation has a greater impact on the primary metric, more visitors are exposed to that variation. This is useful to capitalize on conversions during a short cycle like a weekend sale.
With a contextual bandit, you can personalize the user experience and ensure the variation chosen best fits the visitor's context. If the visitor has spent much money on air travel in the last six months, they would likely get the travel credit card offer. A user with a high-interest car loan likely sees the auto loan refinance offer. A user who recently opened a small business likely sees the business card offer.
Other use cases for contextual bandits include the following:
- Personalized offers – Personalize the offers users see based on their attributes, such as products owned or viewed, to increase conversion rates.
- Members-only sale – Deliver variations based on the user's membership for a members-only sale or something similar.
- Time-based content delivery – Increase user engagement during specific times of day by showing different types of content or offers, such as morning promotions and evening discounts.
- Location-based offers – Target a user's location with region-specific promotions or discounts to increase conversion rates in different regions.
- Behavior-triggered messaging – Show different messages or call-to-actions based on recent behavior, such as reminders to complete a purchase or suggestions for related products, to reduce cart abandonment rates and increase completed purchases.
- Dynamic pricing strategies – Use different pricing strategies or discounts based on a user's browsing history, purchase intent, or location to optimize pricing and maximize revenue while remaining competitive.
Note
Contextual bandits do not generate statistical significance. Instead, they push traffic to variations that match user attributes. See Contextual banditsContextual bandits for information.
Mathematical difference between Stats Accelerator and a multi-armed bandit
In traditional A/B/n testing, a control schema is defined in contrast to several variants to determine if they are better or worse than the control. Typically, such an experiment is done on a fraction of web traffic to determine the potential benefit or detriment of using a particular variant instead of the control.
Suppose the absolute difference between a variant and control is significant. In that case, only a few impressions of this variant are necessary to confidently declare the variant as different (and by how much). On the other hand, when the difference is small, more impressions of the variant are necessary to spot this slight difference.
The goal of Stats Accelerator is to spot the significant differences quickly and divert more traffic to those variants that require more impressions to attain statistical significance. Although nothing can ever be said with 100% certainty in statistical testing, Optimizely guarantees that the false discovery rate is controlled, which bounds the expected proportion of variants falsely claimed as having a statistically significant difference when there is no actual difference (users commonly specify to control the false discovery rate at 5%).
In a nutshell, use Stats Accelerator when you have a control or default and investigate optional variants before committing to one and replacing the control. With Multi-Armed Bandit, the variants and control (if they exist) are on equal footing. Instead of reaching statistical significance on the hypothesis that each variant is different or the same as the control, Multi-Armed Bandit attempts to adapt the allocation to the variant with the best performance.