Optimizely's automatic sample ratio mismatch detection

A sample ratio mismatch (SRM) occurs when the traffic distribution between variations in a Stats Engine A/B experiment becomes severely and unexpectedly unbalanced, often because of an implementation issue or third-party bots.

If an SRM does occur, it indicates a potential external influence affecting the traffic distribution. Exercise caution and do not overreact to traffic disparities. An imbalance does not automatically signify that an experiment is useless.

Optimizely Experimentation alerts you to experiment deterioration promptly. Early detection helps you decide the severity of the imbalance and stop a faulty experiment. This early detection can greatly reduce the number of potential users exposed to a faulty experiment.

To rapidly detect deterioration caused by mismanaged traffic distribution, Optimizely Experimentation's automatic SRM detection uses a statistical method called sequential sample ratio mismatch (SSRM). Optimizely's SSRM algorithm continuously checks traffic counts throughout an A/B experiment. It provides immediate detection at the beginning of an experiment's lifecycle instead of waiting until the experiment's end to test for an imbalance.

Going through your old experiences and trying to find imbalances using an online ratio mismatch calculator is not helpful. This retroactive or end-of-experiment imbalance check is not a recommended use of your time. Retroactive imbalance testing informs you about a possible implementation problem only after the experiment has collected all the data, which is far too late and goes against why most experimenters want imbalance detection in the first place. In another, possibly uncommon scenario, an SRM might occur during the experiment. However, by the conclusion of the experiment, the total traffic across variants becomes similar, effectively concealing the SRM due to the influx of additional data. 

You should run automatic checks. The algorithm Optimizely created checks for imbalances daily (not just at the end of an experiment) so you can identify problems early.

Optimizely Experimentation does not check for visitor imbalances when you segment your results.

The Sure Thing Principle (STP)1,2 states that if an action increases the chance of a negative outcome across segments, it increases the chance overall. So, Optimizely Experimentation does not check for visitor imbalances in segments.

Citations: 1. Savage, L. J. (1954). The Foundations of Statistics. John Wiley & Sons. 2. Joyce, J. M. (1999). The Foundations of Causal Decision Theory. Cambridge University Press.

Optimizely Experimentation does not check for visitor imbalances for the following:

An SRM occurs when the traffic distribution between variations in a Stats Engine A/B experiment becomes significantly imbalanced. Optimizely Experimentation's Stats Engine does not generate SRMs, and its traffic-splitting mechanism is trustworthy. A severe traffic distribution imbalance may lead to experiment degradation and, in extreme cases, inaccurate results.

For example, in a Stats Engine A/B test, you set a 50/50 traffic split between Variation A and Variation B. But instead, you observe a 40/60 traffic distribution.

Evaluating experiments for traffic imbalances is most helpful at the start of your experiment launch period. Finding an experiment with an unknown source of a traffic imbalance lets you turn it off quickly and reduce the blast radius. Furthermore, performing an A/A test before running an actual experiment can help uncover potential issues in the configuration, thereby reducing the likelihood of encountering an SRM. 

For information on why a traffic imbalance may be occurring, see Possible causes for traffic imbalances.