Create an advanced experiment plan and QA checklist
- Optimizely Web Experimentation
- Optimizely Personalization
- Optimizely Performance Edge
- Optimizely Feature Experimentation
- Optimizely Full Stack (Legacy)
Note
This article is part of The Optimization Methodology series.
A full experiment plan gathers decisions from stakeholders into a single, collaborative document. It provides a detailed summary of the motivations and mechanics of your experiment or campaign. When shared, it helps build visibility across the organization and scale your experimentation strategy by establishing a standardized process.
A QA checklist builds efficiency around your QA process to ensure your experiments and campaigns work as intended.
Materials to prepare
- Variation descriptions (wireframes or screenshots)
- Summary of all technical and design assets needed for the experiment
- Parameters for significance and lift that indicate that the change will be implemented permanently
People and resources
- Program manager
- Developer
- QA team
- Designer
Actions to perform
- Create a test plan document.
- Create a rigorous QA checklist.
- Review and update plan with stakeholders.
- Confirm scope of test.
- Confirm stakeholders who will create required resources.
- Document responsibilities and deadlines (in Kanban, Gantt chart, or another internal method).
- Finalize test plan.
Deliverables
- Test plan document containing:
- Details for building an experiment
- Technical requirements
- Scope of the experiment
- Creative assets or wireframes
- Screenshots of variations
Watch out for:
- Ill-defined scope
- Lack of true hypothesis or goals
- Lack of executive buy-in
- Missing screenshots
- Poor understanding of resource needs
- Inaccurate effort estimates
- Inadequate documentation for QA
- Plan not being shared with the proper stakeholders
- Lack of adherence to plan when building the test
Note
If your team is starting to run its first tests, see the basic experiment plan.
Experiment design document template
Your experiment design document should include:
- The actual code used for implementation
- Alignment with sprint planning
- Experiment ID
- Roles/responsibilities
- Primary, secondary, and monitoring goals for the experiment
- Correlating goals to business value
You may also want to use minimum detectable effect to choose what type of test to run.
QA checklist template
When you plan an experiment, include the QA team so they can create a checklist. This helps build a rigorous, efficient process for cases that need to be checked and eliminates those that do not.
Your QA checklist should include:
- Goals or events in the experiment and how each is triggered
- Added functionality
- Common use cases including expected user flows to and from the page
- Included or excluded audiences
- URLs where the experiment should run
- Sample workflow to fire a goal (especially for custom events)
Your QA team should grade each use case as pass or fail. Push the experiment live once everything on the list has passed the test.