Mutual exclusion overview
- Optimizely Web Experimentation
- Optimizely Personalization
- Optimizely Performance Edge
- Optimizely Feature Experimentation
- Optimizely Full Stack (Legacy)
Mutual exclusion is a mechanism to ensure that a single user is not exposed to multiple experiments simultaneously, especially when those experiments test the same algorithm or could cause interaction effects. This helps in obtaining clear and untainted experiment results.
Interaction effect
By default, experiments in an Optimizely Experimentation project can overlap, meaning a single user might be exposed to multiple experiments. This can lead to an "interaction effect" where the results of one experiment are skewed by the overlap with another. An interaction effect occurs when users exposed to multiple overlapping experiments behave differently, skewing the results for individual experiments. To minimize this risk, some or all experiments can be made mutually exclusive. You are more likely to see interaction effects when
- Testing the same area of the application or page.
- Testing the same user flow (for example, a checkout funnel or multi-step form).
- Testing shared conversion metrics.
If your tests run on different areas of your application or track different goals, they are less likely to have an interaction effect.
When to use mutual exclusion
You should implement mutual exclusivity or sequential experiment execution when there is a high risk of interaction effects due to overlapping functionality or metrics critical to decision-making. This requires evaluating risk tolerance for experiment overlap and planning variation designs, goals, and execution schedules.
Consider using a mutually exclusive experiments or running sequential tests (waiting for one to end for the next to start) when the following are true:
- Testing elements on the same page with the same goal – Evaluate whether to run simultaneously, sequentially, or as a single multivariate experiment.
- Testing subsequent pages of the funnel with the same goal – Consider simultaneous, sequential (bottom-up), or a multi-page experiment.
When not to use mutual exclusion
If none of the previous conditions (same application area, user flow, or shared conversion metrics) apply, creating mutually exclusive experiments is usually unnecessary, as overlapping experiments proportionally expose variations with minimal interaction risk.
Best practices
- Make experiments mutually exclusive only when necessary to avoid unduly restricting traffic.
- Experiments within an exclusion group must be part of the group from the start of the first experiment until the end of the last to maintain fixed traffic allocation and visitor bucketing
- Do not add experiments to an exclusion group after one has started, as this can shift traffic allocation and expose visitors to multiple experiments.
- Do not remove running experiments from an exclusion group or reallocate traffic in ways that could introduce bias or overlap.
Configure mutual exclusion
To configure mutual exclusion in Experimentation, see the following articles:
- Feature Experimentation – Mutually exclusive experiments in Feature ExperimentationMutually exclusive experiments in Feature Experimentation
- Web Experimentation – Mutually exclusive experimentsMutually exclusive experiments
- Personalization – Mutually exclusive experimentsMutually exclusive experiments
- Performance Edge – Mutually exclusive experimentsMutually exclusive experiments