Run a multi-armed bandit optimization in Feature Experimentation
- Optimizely Feature Experimentation
You might want to run a test focusing on maximizing conversions from your variations instead of finding the variation most likely to perform consistently better than your baseline. A multi-armed bandit (MAB) optimization is a different type of experiment, compared to an A/B testA/B test, because it uses reinforcement learning to allocate traffic to variations that perform well while allocating less traffic to underperforming variations.
MAB optimizations do not generate statistical significance. Instead, the algorithm pushes traffic to variations with the most conversions; the reason for a variation's performance is unimportant.
MAB is for optimization, not experimentation. Its tests are best suited for maximizing conversions during short, temporary experiences such as headline testing or a holiday weekend sale. You should never use MAB tests for exploratory hypotheses or variation selection.
MAB's primary goal is to answer: Which variation gets us the largest reward? where the "largest reward" is the highest revenue or most conversions. For information on MABs, see how to Maximize lift with multi-armed bandit optimizationsMaximize lift with multi-armed bandit optimizations.
Best use cases
The following cases may be a better fit for a multi-armed bandit optimization than a traditional A/B experiment:
- Promotions and offers – Users who sell consumer goods on their site often focus on driving higher conversion rates. One way to do this is to offer special promotions that run for a limited time. Your changes will not be permanent, and an MAB optimization will send more traffic to the over-performing variations and less traffic to the underperforming variations for the duration of the promotion.
- Headline testing – Headlines are short-lived content that loses relevance after a fixed time. If a headline experiment takes as long to reach statistical significance as the lifespan of a headline, then insights gained from the experiment are irrelevant. Therefore, an MAB optimization lets you maximize your impact without balancing experiment runtime and the natural lifespan of a headline.
- Webinars – You can boost registration for webinars or other events by experimenting with several different versions of calls to action to sign up for your webinar.
For algorithmic details of MABs at Optimizely, see the end-user documentationend-user documentation.
Configuration overview
To configure a MAB, complete the following:
- (Prerequisite) Create a flagCreate a flag in your Feature Experimentation project.
- (Prerequisite) Handle user IDsHandle user IDs.
You should configure a user profile serviceuser profile service to ensure consistent user bucketing if you are using a server-side SDK.
- Create and configure a MAB rule.
- If you have not done so yet, implement the Optimizely Feature Experimentation SDK's Decide method in your application's codebase through a flagflag.
- Test your MAB rule in a development environment. See Test and troubleshootTest and troubleshoot.
- Discard any test user events and enable your MAB optimization rule in a production environment.
Create an MAB optimization in your Feature Experimentation project
To create an optimization in the Optimizely app, complete the following:
- Go to Flags, select your flag, and select your environment.
- Click Add Rule and select Multi-Armed Bandit.
- Configure your MAB rule.
- (Optional) Search for and add audiences. To create an audience, see Target audiencesTarget audiences. Audiences evaluate in the order in which you drag and drop them. You can match each user on any or all of the audience conditions.
- Set the Traffic allocation slider to set the percentage of your total traffic to bucket into the experiment.
- Add metrics based on tracked user events. See Create eventsCreate events to create and track events. For information about selecting metrics, see Choose metricsChoose metrics. For instructions on setting up metrics, see Create a metric in Optimizely using the metric builderCreate a metric in Optimizely using the metric builder.
- Choose the variations you want to optimize. Unlike A/B experiments, you do not need to compare to a baseline experiment because statistical significance is not calculated with MAB optimizations. See Why MABs do not use a baseline.
- (Optional) Add the MAB to an Exclusion GroupExclusion Group.
- Click Save.
If you plan to change the traffic allocation after starting the experiment, then implement a user profile service. See Ensure consistent user bucketingEnsure consistent user bucketing. Also, create a user profile service if you plan on using Stats Accelerator.
- Click Run on the MAB rule. If it is not already running, click Run on the ruleset (flag).
Implement the MAB
If you have already implemented the flagimplemented the flag in your application's codebase, no further configuration is required for the flag delivery. If you have not, implement the Decide method call in your code to enable or disable the flag for a user. See the Run a multi-armed bandit optimization Run a multi-armed bandit optimization in the developer documentation for code samples.
See the following SDK reference documentation for more detailed examples.
- Android example usageAndroid example usage
- Go example usageGo example usage
- C# example usageC# example usage
- Flutter example usageFlutter example usage
- Java example usageJava example usage
- JavaScript example usageJavaScript example usage – SDK versions 6.0.0 and later.
- JavaScript (Browser) example usageJavaScript (Browser) example usage – SDK versions 5.3.5 and earlier.
- JavaScript (Node) example usage – SDK versions 5.3.5 and earlier.
- PHP example usagePHP example usage
- Python example usagePython example usage
- Ruby example usageRuby example usage
- Swift example usageSwift example usage
Optimizely Feature Experimentation uses the Decide method call to decide whether a user qualifies for the rule and which variation they receive. The Optimizely Feature Experimentation SDKs let you reuse the exact flag implementation for different flag rules.
Remember, a user evaluates against all the rules in a ruleset in order before being bucketed into a rule's variation. See Create feature flagsCreate feature flags.
MAB results
Because MAB optimizations do not generate statistical significance, their results page differs from an A/B test. Instead of calculating statistical significance, MABs push traffic to variations with the most conversions; the reason for a variation's performance is unimportant.
For information, see the user documentation on the Optimizely Experimentation Results pagethe Optimizely Experimentation Results page.