Guide your program with an experimentation charter
- Optimizely Web Experimentation
- Optimizely Personalization
- Optimizely Performance Edge
- Optimizely Feature Experimentation
- Optimizely Full Stack (Legacy)
Note
This article is part of The Optimization Methodology series.
When you have Optimizely Experimentation ready to start running experiments, consider how you can make the most effective use of the platform. If you immediately jump into testing, you may face the following:
- Inefficient test planning and configuration.
- Tests launched with errors or mistakes, or missing goals or audiences.
- Low velocity.
- Poor communication between team members.
- Tests that generally have no impact.
- An inability to effectively learn from test results.
Plan before you start testing to avoid these scenarios. A charter helps you track the mission, resources, and workflow that govern your A/B testing strategy.
Materials to prepare
Address the following questions before creating your testing charter:
- Business requirements – What are the core responsibilities of the optimization team?
- Budget – What resources do you have?
- Performance benchmark data – What time investments are required to meet performance goals?
- Executive sponsorship – Do you have approval for resources, including time investments?
- Team proficiency – What experience and skill sets do available team members have?
People and resources
- Executive sponsor.
- Program manager.
- Team leads (technical, design, analytics).
- External consultant, if needed.
Actions you will perform
- Create a testing charter.
- Map a workflow.
- Budget time and resources.
- Secure approval from stakeholders.
Deliverables
- A testing charter that documents roles and responsibilities, including the following:
- Role definitions.
- Time (in percent) spent per role.
- Program and individual metrics.
- Optimization workflow
- Inputs and outputs for each step of the workflow
- Responsibilities and contributions by step
- Evaluation of the team's skill sets against program initiatives and goals
Watch out for
- Difficulty establishing roles and responsibilities before creating a well-articulated workflow.
- Challenges to establish staffing requirements and time investments before you have a prioritized roadmap of tests and campaigns.
What to include
Your testing charter may include a brief mission statement, but it should include details to help anyone new or unfamiliar with the testing program learn more. You do not need to include everything from the following list, but you should be thorough for your team.
The testing charter should be a working document owned by the program manager (or equivalent). One or two months into the testing program, the program manager should revisit the document to see how the actual process compares to this plan and make any appropriate modifications. The program manager can then review the charter quarterly or semi-annually.
To get started, download the testing charter template (.rtf). You can copy and modify this template in the program of your choice. Here are the explanations for each section of the document:
- Introduction – Set the context and define the purpose for the rest of the charter. Why did you purchase the Optimizely Experimentation testing platform? What problems were you trying to solve? What are the long-term goals and expected outcomes?
- Team – Define the testing team and include a description of their roles defined by the Optimizely Experimentation platform privilegesroles defined by the Optimizely Experimentation platform privileges.
- Process – Create a workflow diagram in Keynote or PowerPoint from conception, execution, and results-sharing to (potentially) onsite implementation. Add a brief description and a list of key participants for each of the steps you outline.
- Meetings and meeting cadence – Decide when to have brainstorming and status meetings to check on upcoming tests and review ongoing or completed ones. Decide how often the testing team meets with a broader group.
- Test Velocity – Set a goal for testing velocity and modify it as needed based on the actual testing pace at your company. You may want one test always running, ten tests per week, or twenty-five per quarter.
- Test plan and results templates – Create a link to the most updated testing template the company uses.
- Links key – Link to all other essential information, including your current testing roadmap, templates (described previously), and an archive of test results. If team members bookmark this page, they should be able to find any other relevant information they need. You may also want to link to pages that document definitions of the following:
- Test goals – For example, Purchase Confirmation – A pageview goal targeted to http://www.example.com/thank_you_page.
- Targeting conditions – For example, Product Pages – Targeted with a substring match to example.com/productID.
- Audiences and dimensions – For example, Category Affinity defines users who are interested in a particular category and is based on a recent product or category pageview associated with that category.
- Other guidelines – Decide what guidelines and rules your team follows when it comes to testing. Do different types of tests have different processes for QA and approval? Are there standard goals included in every test? When is a winning variation served through the Optimizely Experimentation platform, and when is it hard-coded to the site?