Best practices: From research to hypothesis creation
- Optimizely Web Experimentation
- Optimizely Personalization
- Optimizely Performance Edge
- Optimizely Feature Experimentation
- Optimizely Full Stack (Legacy)
Note
This article is part of The Optimization Methodology series.
Generating a data-informed hypothesis is one of the most important steps in experience optimization. The most effective experiments start with questions to identify areas of uncertainty about your site or examine assumptions.
Materials to prepare
- Idea submission form
People and resources
- Program manager (facilitator)
- Designer
- Marketers
- Analysts
- Merchandiser
- Developer
Actions to perform
- Meet with a broad team.
- Translate data, observations, and insights into questions.
- Turn questions into hypotheses.
- Set a goal for the number of new ideas.
- Document ideas.
Deliverables
- List of hypothesis statements, including:
- Experiment name
- Hypothesis
- Variation description
What to watch out for
- Producing only a small number of ideas
- Producing ideas that are similar (not creating a variety of tactics)
- Producing ideas that are too similar or too difficult on average
- Producing a list that focuses on one or two pages of the site instead of the full funnel
Prepare with research
Use a business intelligence report and a product roadmap during your ideation. Previous experiment results, site analytics, industry research, and your product roadmap provide a detailed view of customer experience and expectations. Examine assumptions about site experience and generate questions to answer through testing.
Bring key insights to the meeting, not raw data. Brainstorming sessions can get sidetracked by undigested data. Analyze your data first to avoid this. Find pain points, anomalies, and interesting moments, such as the following:
- Top trafficked pages
- Conversion CTAs on your top pages
- Top user flows
- Drop-offs at each stage of those user flows
- Bounce rate on each page of the site
Use these insights and your business intelligence report to focus on your opportunities.
You can also create an idea submission form for employees to submit their ideas for testing and personalization. This lets you gather different perspectives, promote data optimization, and develop ideas.
Brainstorm
Here are a few guidelines for productive brainstorming.
Ask a board team
Gather perspectives from different stakeholders to help identify problems to test. The board team can consist of the following people:
- Stakeholders who are directly involved in testing
- Analysts who work with your site’s data
- Developers involved in creating features
- Designers who create assets for the site
- People at the company who see your visitors at a different stage of the customer journey
A holistic view of the customer life cycle can help during brainstorming. For example, if some visitors go to your site from an email campaign or Google ad, your SEO or content marketing manager may have advice on effective value propositions for the site. Customers may purchase on your site and then work with an account manager. The account manager, whose key priorities are retention and loyalty, can help brainstorm ideas to optimize for repeat purchases.
Meet in person
In-person meetings promote cross-functional context sharing, creating a rich environment for ideas.
Meet monthly
Meet on a frequent and regular cadence. For some optimization programs, this session also presents an opportunity for a retrospective of experiment results. Monthly meetings help you incorporate your knowledge into ideation to prioritize areas to test next.
Prioritize opportunities
Bring a prioritized list of the site pages that most impact your business goals. Frame your brainstorming session to focus on each page in order of priority.
Generate questions
Focus each session on generating 5-10 questions to evaluate through testing. Ask productive questions and examine untested assumptions about your site. This helps identify the cause of problems, instead of just alleviating certain frustrations on the site. Consider your business model, visitors, and product roadmap to find opportunities to optimize.
At the end of your ideation session, you should have:
- Questions or assumptions to evaluate through testing or personalization – Organize this list by location or where the test or campaign should focus, such as the home page, product details pages, or checkout funnel.
- A description of your target visitors – Are you evaluating first-time visitors to the site or customers who make large purchases? Identify specific audiences to target when you test or personalize the site.
Defer judgment
Focus on generating questions not answers. Participants should not feel pressured to design tests or present changes. Explore the “why.”
When you revise your idea into a complete hypothesis and create an experiment plan, consider what changes to make to the site experience to optimize for certain visitor behaviors.
Create a hypothesis
After your brainstorming session, turn your questions into measurable hypotheses.
Strong hypothesis statements help you turn data and insights into focused proposals for action.