Design an effective hypothesis

Strong hypothesis statements help you turn data and insights about visitor behavior into focused proposals for action. Each hypothesis is an idea to be tested; every idea confirmed or rejected informs you about visitor expectations and behaviors to improve optimization. Consistent hypothesis-driven experimentation helps your program make an impact on company goals. Learn more with the Optimizely Academy courses.

For example, you notice a high abandonment rate in your purchase funnel. You hypothesize that links in the funnel distract visitors, so you experiment with removing them. The improvement in completed purchases is more than just a lift. It confirms your hypothesis and understanding of your visitors and their site experience. Use this insight to help decide what to optimize next, why, and how to measure results.

Programs that do not use hypotheses risk wasting resources on unfocused experimentation that fails to make a business impact.

Download the hypothesis worksheet (.pdf) to start.

A hypothesis is a prediction you create before running an experiment. The common format is:  If [CAUSE], then [EFFECT], because [RATIONALE].

Strong hypotheses consist of three distinct parts: a definition of the problem, a proposed solution, and a result.

Problem

Experience optimization should solve a problem in the customer experience. When defining a hypothesis, start with a meaningful problem: an issue in visitor experience that you would like to solve.

Use qualitative and quantitative sources to validate your problem. Use data to confirm the issue rather than rely on an assumption. 

Explain the problem from the visitor's perspective to better understand it and generate hypotheses about its root cause.

Problem definition – Users do not see the filters on the search results page.

Data validation – Less than 15% of users use filters when searching for products, which is low compared to industry standards.

Propose a solution. Describe the change so someone who reads the hypothesis can understand the change without screenshots. Add a rationale that explains why this solution is the right one to solve the identified problem.

Proposed solution – Move the filters to the left side of the results.

Rationale – This is the most common filter location; users are more likely to notice them there.

Predict a result that connects your hypothesis with key business metrics. Include metrics that determine the success or failure of your experiment. Decide on the specific metrics to track success in the experiment or campaign.

Primary metric – % of users who use filters increases.

What success looks like – % of users who move on to a product page increases; the purchase rate increases.

Example 1

Example 2 

The difference between a well-formulated hypothesis and a guess is data. Build a business intelligence report. Carefully observe your customer’s journey through your site with direct data sources like web analytics data and indirect data sources like competitor overviews.

Use data linked to your company’s goals to ensure you focus on areas of impact rather than making UX changes in isolation. Data sources can help you validate the problem and the solution rationale. Avoid generating hypotheses based on intuition.

To efficiently test your hypothesis, identify the metrics to track and define clear criteria for success and failure. 

For example, you hypothesize that removing breadcrumb navigation from the checkout page will help visitors stay in the funnel and increase conversions. The difference between the original and the variation is the presence or absence of breadcrumbs. The effect of that change can be measured in the number of conversions.

Hypothesis-driven experimentation gives you insight into visitor behaviors. These insights generate additional questions about your visitors and site experience and drive an iterative learning process.

The cycle should follow this general pattern:

Begin your process with a problem, not a solution. If a solution fails to deliver the expected result, use the problem statement to explore potential solutions and iterate. This practice lets you focus on the problems that prevent your company from reaching its goals.

Learn about connecting your optimization program to company goals.