Section rollups in multivariate tests for Optimizely Web Experimentation

In a multivariate test (MVT) in Web Experimentation, using section rollups lets you see how each variation of a section performed against the others, summing across all the combinations it appears in. 

Section rollups let you

For example, a MVT has two sections (1 and 2) and two variations (A and B). This would generate the following combinations:

Or explained in a table

If you roll up on Section 1, you can see a comparison for combinations 1 and 2 against combinations 3 and 4. From there, Optimizely measures the overall conversion rate of 1A versus 1B, and averages across the different values for Section 2.

Optimizely treats these section rollup A/B tests just like any normal A/B test created outside of an MVT. It calculates significance and confidence intervals in the same way and applies the same False discovery rate control to account for multiple variations (and multiple metrics if applicable) within that section.

The main difference between a section rollup A/B test and an A/B test on that section run in isolation is that interaction effects can appear in the section rollup A/B test results.

Interactions are effects that occur only in the presence of two or more combinations together. They can be positive or negative, but either way can make it more difficult to understand what causes the lift in a rollup view.

For example, suppose that you are using a MVT to test a button’s text color in one section and the button’s background color in another section at the same time. If you test black variations for both text and background simultaneously, the combined effect can have a stronger negative effect than the effects of each variation on its own.

The interaction effect is the additional effect from the two black variations appearing together.

If you run an A/B test on text color alone, the improvement reported on the experimentation results page shows the expected gain in conversions from changing the text color to black.

However, the section rollup A/B test improvement should not be interpreted this way. Since many visitors in that test came from the MVT combination showing black text against a black background, their conversion rate is much lower than expected in cases where the text is black, but the background is not.

Because this effect is absent in the control for the text color section, this would cause the improvement (of a black text over the control text color) reported at the rollup level to be biased downwards.

In other words, the negative response to black text on a black background will make it appear that you should expect a smaller improvement from changing the text color to black than you are likely to get.

To help protect yourself against interaction effects unduly influencing your business decisions, Optimizely recommends a section rollup workflow that begins by looking at MVT combinations and proceeds to section rollups.

To determine whether strong interactions exist, start by comparing the estimated improvement from the variations in the section in question at both:

For example, say you are working with a MVT containing three sections, each with two variations.

In this example, you want to make a decision based on the rollup for the button copy section. To determine if there are interaction effects in your sections, compare the improvement shown in the button copy section rollup to the metrics shown for combinations 1 and 5 (the combinations where both button background color and button text color use their original colors).

If the MVT is free of interactions, the estimated improvement from these two levels should be close. If they are not—especially when the sample size within each MVT cell is large—then that should be taken as a sign that interactions are present, and you should be cautious in interpreting results at the section rollup level.

If there are no interaction effects, then the improvement reported in the section rollup is equivalent to the improvement that you would see from running this A/B test separately. The improvement you actually observe from deploying the variation should closely match the improvement reported on the results page, regardless of what variations you deploy in the other sections.

If there are interaction effects, then the improvement reported in this section rollup is essentially an average over all the candidate variations in the other sections. The improvement you actually observe from deploying the variation may or may not deviate greatly from the improvement reported on the results page, depending on exactly which combination of variations you deploy from the other sections.

To see the results of an experiment broken out on the section rollup level, follow these steps:

The results for the rollup you selected display.