Controlled traffic allocation ramping

Ramping is when you gradually expose traffic to new test variations. The process can introduce inefficiency and risk if you ramp your traffic carelessly. Controlled ramping of your traffic allocation lets you mitigate risk without compromising data accuracy or time to statistical significance. 

The controlled ramping procedure is optional and meant as guidance for the highly risk-averse and statistically savvy experimenter. Ramping is optional for launching successful programs of experiments with the Optimizely Experimentation platform.

Controlled ramping has two stages:

The short ramp stage looks for experimental bugs then moves to the maximum power stage quickly. Examples of experiment bugs includes:

You can increment traffic allocation as much as you like (from 1% to 5%) as long as you do it very quickly (as in 24-48 hours). During the preliminary short ramp stage, you must not wait for statistical significance to proceed to the maximum power ramp stage. You should also avoid continuously monitoring for significance during the short ramp period.

See Recommended controlled traffic ramping process for Optimizely's recommendations.  

The maximum power stage looks for a winner between variations while balancing speed and precision. After the preliminary short ramp stage, to start the maximum power stage you should:

Optimizely Experimentation recommends starting the experiment in the short ramp stage with a 10% traffic allocation and a 50/50 variation split.

Once you move to the maximum power stage and launch the experiment, reset your results, then set the traffic allocation to 100%, but keep the same 50/50 variation split.

Once your experiment concludes, you should implement the winning variation into your code. It is important that you do not simply change the traffic distribution to the winning variant in this experiment.

Although it is not recommended, you can implement the variation as a hotfix in the Optimizely app by doing the following,

Here is an example of what Optimizely recommends for a controlled ramping experience using an Optimizely Web Experimentation A/B test project.

First, during the short ramp stage, set the traffic allocation to 20% with a 50%/50% split between the control (labeled Original) and the variation:

Let the A/B test run with a 20% allocation for a maximum of one to two days. During this time, verify that the variation is running correctly. If you notice something incorrect with the variation, stop the A/B test immediately to stop exposing customers to a negative experience right away.

If the variation behaves correctly, transition to the maximum ramp stage to test for a winner between the control and the variation. To do so, pause this experiment, reset results, and launch the new experiment by setting traffic allocation to 100%:

At the conclusion of your experiment, implement the winning variation into your code. It is important that you do not simply change the traffic distribution to the winning variant in this experiment.

Although it is not recommended, you can implement the variation as a hotfix in the Optimizely Web by doing the followingfollowing:

Here is an example of what Optimizely recommends for a controlled ramping experience using an Optimizely Feature Experimentation project.

First, during the short ramp stage, set the traffic allocation to 20% with a 50%/50% split between the control (labeled the Off variation) and the variation:

Let the A/B test run with a 20% allocation for a maximum of one to two days. During this time, verify that the variation is running correctly. If you notice something incorrect with the variation, stop the A/B test rule immediately to stop exposing customers to a negative experience right away.

If the variation is behaving correctly, transition to the maximum ramp stage to test for a winner between the control and the variation. To do so, reset your results, then launch your experiment by setting traffic allocation to 100%:

At the conclusion of your experiment, you should implement the winning variation's changes in your code. Although it is not recommended, you can implement the variation as a hotfix in the Optimizely app by doing the following,

Although it does not hurt your experiment validity to initially set your experiment to a low traffic allocation (for example, below 10%) and adjust later, it is costly to keep an experiment running for more than a few days at a low traffic allocation. The experiment will not have sufficient power to detect any meaningful impact. Without sufficient power, you risk placing users in a bad experience for too long, which may cause permanent user churn.

Optimizely recommends avoiding the following poor practices: