Experiment types in Web Experimentation

Optimizely Web Experimentation and Optimizely Performance Edge provide three different experiment types: A/B testing, multivariate testing, and multi-page (funnel) testing. Learn their differences and evaluate the benefits and drawbacks of using each type.

See:

Campaigns, experiments, experiences, and variations are terms related to Optimizely Experimentation testing. In general, campaigns have experiences, and experiments have variations. 

A/B testing, also known as split testing, is a method of website optimization in which the conversion rates of two versions of a page—version A and version B—are compared to one another using live traffic. Site visitors are bucketed into one version or the other. 

By tracking how visitors interact with the page—the videos they watch, the buttons they click, or whether or not they sign up for a newsletter—you can determine which version of the page is most effective. 

A/B Testing is the least complex method of evaluating a page design and is helpful in various situations.

One of the most common ways of A/B testing is to test two different design directions and compare them with each other. 

Example:

The current version of a company's home page might have in-text calls to action, while the new version might eliminate most text but include a new top bar advertising the latest product. After sufficient visitors are funneled to both pages, the number of clicks on each version of the call to action can be compared.

Even though many design elements are changed in this kind of A/B test, only the impact of the design on each page's business goal is tracked, not individual components.

A/B testing can also be used as an optimization option for pages where the effectiveness of only one design element needs to be evaluated.

Example:

A pet store that runs an A/B test on their site might find that 85% more users sign up for a newsletter when it is displayed as held up by a cartoon mouse than one that emerges from a boa constrictor's coils. In A/B testing, a third or fourth version of the page can be included in the test, which is called an A/B/C/D (or A/B...n) test. This means that traffic to the site must be split into thirds or fourths, with a lower percentage of visitors visiting each site.

Simple in concept and design, A/B testing is powerful and widely used.

Using a small number of tracked variables lets these tests deliver reliable data, as they do not need much traffic. This is helpful if your site has a small number of daily visitors.

A/B testing is easy to interpret and some large sites can use it as their primary testing method to run cycles of tests one after another instead of more complex multivariate tests.

A/B testing is best used to measure the impact of two to four variables on interactions with the page. Tests with more variables take longer to reach statistical significancestatistical significance, and A/B testing does not reveal any information about the interaction between variables on a single page.

An A/A test is an experiment in which two identical versions of a webpage or application are tested against each other. Unlike an A/B test, which compares a control version with a variant to determine which performs better, an A/A test compares two identical versions to ensure that the testing platform and setup are functioning correctly. This type of test is also known as calibration testing.

The primary purpose of an A/A test is to validate the experiment setup and ensure the reliability and accuracy of the data collected. By running an A/A test, you can identify any issues with the experimentation platform or the implementation of the test before conducting actual A/B tests. This helps in ensuring that the platform is ready for widespread use and that the data captured during experiments is valid.

Data validation – Verify the accuracy of data capture and the test setup. For example, metrics that track specific user actions, such as click events or transactional events, can be included in an A/A test to validate that these custom events are being captured correctly by the experimentation platform.

Baseline measurement – Determine the baseline conversion rate for a specific goal before starting A/B or multivariate tests. This helps in setting realistic targets and measuring the impact of future tests.

Troubleshooting – Identify and resolve any issues with the implementation of the experimentation platform or the test setup. For websites using single-page applications (SPA), an A/A test can be performed whenever the tag is newly implemented into the header code to ensure consistency across all experiences.

Ensure platform readiness – Validate the experimentation platform before conducting actual tests to confirm it is ready for widespread use.

Identify issues early – Detect any issues with the implementation of the experimentation platform or test setup before conducting actual tests, saving time and resources.

Validate data accuracy – Compare identical versions to ensure the data captured during experiments is accurate and reliable.

No real differences – A/A tests compare identical versions, so they do not provide insights into how different variations perform. They are purely for validation purposes.

Potential for anomalies – There is a small chance that the results of an A/A test may show a statistically significant difference due to random chance. This is a probability, not a certainty, and should be interpreted with caution.

Resource-intensive – Running A/A tests requires time and resources, which could otherwise be used for actual A/B tests that provide actionable insights.

Multivariate testing uses the same core mechanism as A/B testing but compares a higher number of variables and reveals information about how they interact with one another. Think of it as multiple A/B tests layered on top of each other.

As in an A/B test, traffic to a page is split between different design versions. Then, a multivariate test measures the effectiveness each design combination has on the business goal.

When a site receives enough traffic to run the test, the data from each variation is compared to find the most successful design and reveal which elements have the most significant positive or negative impact on a visitor's interaction.

Multivariate testing is most commonly done on pages that need several elements evaluated. For example, a page includes a sign-up form, header text, and a footer.

To run a multivariate test on this page, instead of creating a significantly different design as in A/B testing, you can create two different lengths of the sign-up form, three different headlines, and two footers. Next, you can funnel visitors to all possible combinations of these elements.

Testing all possible combinations of a multivariate test is also known as full factorial testingfull factorial testing. It is one of the reasons why multivariate testing is recommended only for sites with a significant amount of daily traffic. The more variations that need to be tested, the longer it takes to obtain meaningful data from the test. It is, however, the most accurate way to run a multivariate test.

For example, varying a page footer may have minimal effect on the page's performance, while running the length of the sign-up form can have a huge impact.

Multivariate testing lets you target redesign efforts to the elements of your page where they have the most impact. This is useful for designing landing page campaigns. For example, the data on the effect of an element's design can be applied to future campaigns, even if the context of the element context changes.

The most significant limitation of multivariate testing is the amount of traffic needed to complete the test. Because experiments are fully factorial, multiple changing elements can result in many possible combinations that must be tested. Even a site with reasonably high traffic can have trouble completing a test with more than 25 combinations.

When using multivariate tests, consider how they fit into your testing cycle and redesign. Even if you have information about the impact of a particular element, you may want to do additional A/B testing cycles to explore other ideas. In some cases, several well-designed A/B tests may be preferable over the extra time needed to run a full multivariate test.

Multi-page (also known as "funnel") testing is similar to A/B Testing, but instead of making variations to a single page, the changes are implemented consistently over several pages. Like A/B testing, site visitors of a multi-page test are bucketed into one version. You can determine which design style is most effective by tracking how these visitors interact with the different pages they see. The key to getting usable data in a multi-page test is preventing users from seeing a mix of variations; they should see a consistent variation throughout a set of pages. This lets one variation be well-tested against another.  

Testing different design directions with each other can be done using multi-page testing. For example, imagine an ecommerce website that lets users search through numerous products, add desired items to a virtual shopping cart, and then purchase the items.  

In this case, users see more than a single page. Instead, they are funneled through several pages before purchasing or leaving the website. Using a multi-page test, you can create two (or more) unique designs for a set of pages. After you do this, ensure that your users see only one design style throughout all the pages.

After enough visitors are funneled through the different designs, the effects of the other design styles can be compared.

Like A/B testing, multi-page testing can provide meaningful and reliable data. It lets users see a consistent set of pages, whether it is the original or a redesigned variation. 

Multi-page testing lets you implement the same changes you make on a single page in a typical A/B test to several pages. This ensures your visitors do not bounce between different variations and designs when funneling through your website.

Multi-page testing has many of the same limitations as A/B testing. Like A/B testing, multi-page testing is best used to measure the impact of only a few variables at a time. Tests with too many variables take longer to run. Determining the effect of each change you make on every page is also more challenging.

When you set up a multi-page test, you must have the same number of variations for every page of the experiment. An uneven number of variations creates inconsistency between pages, worsens the user experience, and makes any data collected challenging to interpret. Additionally, only targeting conditions that apply to all pages in the experiment can be used for multi-page experiments.