Run and interpret an A/A test

When you run an experiment in Optimizely Experimentation in which variation A is identical to variation B, you are running what is known as an A/A experiment, rather than an A/B experiment. Such an experiment is called an A/A experiment because there is no "B" variation. The original page and variation are exactly the same. A/A experiments are also referred to as calibration testing.

An A/A test is different from a monitoring campaign. A monitoring campaign is an experiment without any variations.

In most cases, the majority of your A/A calibration test results should show that the improvement between the identical baseline variations is statistically inconclusive. You should expect this inconclusive result because you made no changes to the original variation. However, when running a calibration test, it is important to understand that a difference in conversion rate between identical baseline variations is possible. The statistical significance of your result is a probability, not a certainty.

As with any experimental process, there is always some percentage of outcomes that turn out to be anomalies because an experiment calculates results on a random sample from the population of all visitors to your experiment. An experiment must make a judgment call on how large a trend indicates a true difference to identify any variation as significant. So, a large enough fake trend can make experiments look like there may be a true difference when none actually exists. The significance level controls this trade-off between identifying trends with significant results and seeing more errors.

Statistical significance is a project-level setting that you can adjust up or down. You can adjust it based on your comfort level with statistical error. This also holds true for an A/A test. Even when there is no difference, there is a small chance that Optimizely Experimentation reports a result based on underlying trends in experiment data.

When you examine the results of your A/A test, you should see the following behavior:

Statistical significance measures how unusual your experiment results would be if there were no difference between your variation and baseline, and the difference in lift was due to random chance alone. When you observe a lift with 90% or higher statistical significance, the observed results are more unusual than expected in 90% of the cases if there is no lift. Assuming no difference in performance between the variation and the baseline, the higher your statistical significance, the more unusual your results seem.

Pay attention to the statistical significance in your experiments, and be skeptical of implementing variations that do not reach your chosen significance level.

With Stats Engine, Optimizely Experimentation accurately represents the likelihood of error, regardless of when you look at your results. Ultimately, you may need to find the happy medium between declaring winning and losing variations with high confidence (thus requiring more visitors) and the opportunity cost of being able to run more experiments.

See Confidence intervals and improvement intervals for information.

Each variation key must be unique to ensure that Optimizely can accurately track and differentiate between the variations being tested. Unique keys prevent data conflicts and ensure the results are correctly attributed to the respective variations.

Events track the success of your experiment by recording specific user actions, such as clicks, pageviews, form submissions, purchases, and scroll depth. 

Create an event to track your changes, to learn more, see Create eventsCreate events.

After creating your A/A test, you should implement the A/A test using Feature Experimentation decide methodsdecide methods. Then, you can start your test.