10 common experiments and how to build them
- Optimizely Web Experimentation
- Optimizely Personalization
- Optimizely Performance Edge
- Optimizely Feature Experimentation
- Optimizely Full Stack (Legacy)
When you start experimenting on your site, you may not know what to test first. You can use these common experiment examples to learn about setting up experiments and explore ideas for your own site.
See Six steps to create an experiment in Optimizely Web Experimentation to build experiments like the ones below.
Note
See how over 35 leading companies execute winning experiment ideas in the Big Book of Experimentation.
These articles provide more information on the different components of building an experiment:
- Hypothesis – Design an effective hypothesis
- Interpret results – Interpret your Optimizely Experimentation Results – Support Help Center
What not to test
Experimentation is both an art and a science. Here are three common mistakes to avoid if you want your tests to make an impact:
- Testing parts of your site that get low traffic – If you have a low-traffic site, see Testing tips for low traffic sites. Avoid running experiments on low-traffic pages. It takes longer to get as many visitors into the experiment as you can get on a high-traffic page, and you may not see statistical significance while the test stalls. This includes remote pages of your site and parts of a page most visitors do not see, like the footer of a high-traffic page or a drop-down list.
- Testing changes that are too small to matter – You may want to test small changes at first because you are not yet sure what impact your testing will have or how visitors will respond. This makes a bold change more valuable because you learn more about your visitors' behavior. A bigger change also means your test can run faster because it makes more of an impact. If you test a change that is too small, you may not see significance, and your program stalls while you wait.
- Testing ideas that are too far removed from core goals – For most programs, what you test on your website relates to three major goals: increasing engagement, collecting data about your visitors (such as name, email, preferences), and purchases. Experiments that do not drive towards the core business goals for your company do not help your program make an impact. Focus your testing on changes that affect metrics that matter.
Idea 1: Optimize for geographical differences
Neha's market research shows that the top concerns of her North American and European customers differ from each other.
- In the U.S., her product's biggest selling point is the flexible, developer-friendly experience it offers.
- In European markets, most sales conversations start out with questions about privacy and security.
Neha wants to experiment with different messaging on her site for European customers and U.S. customers. She thinks that security-focused branding can make her product more compelling for European consumers.
Hypothesis
The product's value proposition focuses on the developer experience, which is a selling point for U.S. customers. However, this misses the top concern of European companies: security. If European visitors see a security-focused value proposition in the hero banner of a high-traffic landing page, the number of leads from that region will increase.
Pages
The URL for the landing page
Audiences
Metrics
- Clicks to View Pricing (primary)
- Clicks to Explore the Docs (secondary)
- Pageviews for the pricing and form pages (secondary)
- Clicks to the form submission button (secondary)
Editor
Original:
Variation:
- Select the variation.
- Click to select the value proposition text.
- Enter a new value proposition under HTML. Click Save.
- QA with the Preview tool.
- When you are ready, click Start Experiment.
Technical skills
None
Interpret results
- Win – Neha's hypothesis is correct. Security-focused value messaging increases conversions from visitors in Europe. She can then expand this branding across the site.
- Lose – The results do not support Neha's hypothesis, and she must determine why. Do visitors in Europe who land on this page still value developer tools over security? Maybe developers, not program managers, make decisions based on this page. She can analyze her data and apply her findings in future tests.
- Inconclusive – Tells Neha that her hypothesis is incorrect, or the test is not distinct enough. Visitors continued to behave in the same way. She could try a more radical change, like switching out all the copy and imagery on the page.
Idea 2: CTAs for new visitors versus subscribers
David manages the landing page for a streaming video site. It features a call to action (CTA): "Join free for a month." All visitors see this page, but research shows that subscribed customers find the button confusing because they have already joined. Because David's company cares about logins, he wants to test a separate CTA for subscribers.
Hypothesis
Changing the Join Free CTA button to match the subscribed visitor's mindset can increase the number of logins by reducing their confusion.
Pages
The URL of the landing page.
Audiences
Metrics
- Clicks to the CTA button (primary)
Editor
Original:
Variation:
- Select the variation.
- Click the text on the CTA to select it.
- Edit or replace the text with your new CTA. To remove extra text, click to select. Under Visibility, click Remove.
- Click Save.
- QA with the Preview tool.
- Click Start Experiment when ready.
Technical skills
None
Interpret results
- Win – The results support David's hypothesis: Returning visitors respond well to a CTA about seeing what is next. Next, he can expand on this theme or test variations that interest visitors. If he has Personalization, he can feature a show that the visitor has watched or highlight popular shows for an audience segment.
- Lose – The results do not support David's hypothesis. He can see if he has made any incorrect assumptions. Are returning visitors looking to do something different from what he thought? Is there something else on the page that influences their decision to log in? A loss can give him insight into how subscribed customers react to this page.
- Inconclusive – suggests that the hypothesis is off, or the change is not enough—visitors continue to behave in the same way. He can try clearer, more distinct messaging or move on to a more urgent experiment.
Idea 3: Remove distractions from the checkout funnel
Jessica's team has a mandate to improve the purchase rate of her company's retail site. They looked at the data and noticed that a high percentage of customers leave the checkout flow through the breadcrumb navigation. Jessica believes that she can improve the purchase rate by removing distractions from the checkout funnel. She wants to validate this idea by testing a less cluttered navigation.
Note
If you are using Optimizely Experimentation to test on a checkout page, see Configure Optimizely Experimentation for PCI DSS compliant use.
Hypothesis
Navigation elements distract customers from completing their purchases. Removing the distractions can increase the purchase rate by 7%.
Pages
Every URL in the checkout flow. You can use substring match if all your checkout pages follow a pattern like yourdomain.com/checkout.
Audiences
Everyone
Metrics
- Clicks to Continue on each step of the checkout (primary)
- Pageviews for the purchase confirmation page (secondary)
- Clicks to Submit Order (secondary)
- Pageviews for each step in the checkout flow (secondary)
Editor
Original:
Variation:
- Select the variation.
- Go to Changes > Create.
- Click Element Change and select the elements you want to remove, such as the container that includes the breadcrumb navigation. Click Hidden to hide the feature without removing the field, or click Remove. Click Save.
- QA with the Preview tool.
- Click Start Experiment when ready.
See Cannot load a password-protected, gated, or session-specific page in the editor if you have trouble loading your page in the Editor.
Technical skills
None
Interpret results
- Win – Jessica's hypothesis is correct. Visitors leave the funnel because they have the opportunity, not because they do not intend to complete their purchase. Jessica can then shorten the checkout flow or remove other distractions from the funnel.
- Lose – The team's assumption about visitors dropping off is not correct; visitors may be leaving on purpose. They may see if there is a negative or confusing experience during checkout.
- Inconclusive – Suggests that the hypothesis is off, or the change is not enough—visitors keep leaving at approximately the same rate. Jessica may investigate where, how, and why visitors leave the funnel.
Idea 4: Rearrange subscription price-points
Emmanuel has noticed that the mid-priced Plus plan on his page receives a surprisingly high percentage of subscribers, compared to the free Basic and upgraded Premium plans. This distribution also matches the click-through rate to the plans details pages. Emmanuel thinks that the position of plan options affects interest and purchases. He wants to test a different arrangement to encourage more customers to evaluate and subscribe to the Premium plan.
Hypothesis
A smaller-than-expected percentage of visitors click the Basic or Premium plans on the pricing page due to positioning. Moving the Premium plans to the center of the page can increase interest and subscriptions.
Pages
The URL for the plans page
Audiences
Everyone
Metrics
- Clicks to Go Premium (primary)
- Clicks to Sign up and Get Plus (secondary)
- Plan details page pageviews (secondary)
- Purchase confirmation pageviews (secondary)
- Revenue (secondary)
Editor
Original:
Variation:
- Select the variation.
- Go to Change > Create.
- Click Element Change and select the container for the highlighted element.
- Click Magnify under Rearrange. Then, click the element you want to rearrange the first element in relation to (the Plus container, in the previous example).
- Use the drop-down list following Rearrange to re-position the element. Click Save.
- QA with the Preview tool.
- Click Start Experiment when ready.
Technical skills
None
Interpret results
- Win – If more visitors click Go Premium in the variation, Emmanuel's assumption is correct that visitors are sensitive to the position of the packages on his pricing page. His results also support the idea that visitors are more willing to consider the Premium package. He can keep experimenting with ways to make the Premium package more visually prominent.
- Lose – The results do not support Emmanuel's hypothesis. He may not have identified the right problem or solution. If his goal is to increase Premium subscriptions, he can investigate what customers like about this package and test ways to draw attention to those value propositions.
- Inconclusive – The change is not big enough, or customers are not very sensitive to the position of the plans. Emmanuel can use this insight in future designs.
Idea 5: Highlight key value propositions
Kate's company offers a number of pricing packages on the site, but research shows that customers find this list overwhelming and have trouble choosing. Based on this information, she wants to experiment with highlighting key value propositions to help customers choose between price points and packages.
Hypothesis
Customers are confused by the packages offered. Highlighting a key value proposition can make it easier for customers to choose, leading to an increase in purchases for that package.
Pages
The URL for the Plans page
Audiences
Everyone
Metrics
- Clicks to the highlighted package (primary)
- Clicks to Add-to-Cart (secondary)
- Pageviews for the package details pages (secondary)
- Pageviews for a purchase confirmation page (secondary)
Editor
Original:
Variation:
- Select the variation.
- Go to Changes > Create.
- Select Insert HTML to add a value proposition next to a package. Insert your text in relation to another element on the page—in this case, the package. Click Magnify, then select the package to highlight.
- Enter your text under HTML. Under HTML Placement, reposition the inserted text as desired.
- Click Save.
- QA with the Preview tool.
- Click Start Experiment when ready.
Technical skills
None
Interpret results
- Win – Kate confirms that highlighting value propositions helps customers choose and purchase a package. She can also check her secondary metric for overall purchases or overall Add-to-Cart clicks to ensure that this plan increases her company's revenue overall. If revenue decreases, she may want to see which plan has a larger revenue percentage and test compelling value propositions for that plan type.
- Lose – The results do not support Kate's hypothesis. She has not identified either the right problem or the right solution. Maybe a different type of value proposition would work better: price per unit if her customers are especially price-conscious, or CTAs for buyer personas if she learns that certain personas have unique package preferences.
- Inconclusive – The change may not be big enough. Kate can consider experimenting her way through a radical redesign of the pricing page.
Idea 6: Symmetric messaging
Wesley's company is advertising a holiday sale on fitness trackers with two different ad campaigns: one focusing on holiday gift-giving and the other on outdoor adventures. His task is to optimize the paid search campaigns. Wesley believes he can quickly increase conversions by customizing the imagery to match each specific ad's messaging, rather than using a generic page for all ads.
Hypothesis
The current landing page for Wesley's site promotes holiday gift-giving. When visitors click ads about a fitness tracker for outdoor adventures, they should see related imagery when they land on the site itself. If he switches the imagery to match the ad the visitor clicked, the site will align with the visitor's mindset and goals, and conversions can increase.
Pages
The URL of the landing page
Audiences
Visitors who match the ad campaign audience condition.
The ad campaign audience condition is sticky, meaning a visitor sees the same experience every time they come back to the site. Visitors who see the adventure imagery will keep seeing mountains. Visitors who see gift-giving imagery will continue to see presents, even if they visit the page without the query parameter.
Metrics
- Clicks to Shop the Sale (primary)
- Clicks to product detail page below (secondary)
- Clicks to top-level navigation (secondary)
Editor
Original:
Variation:
- Select the variation.
- Click the image to select the container.
- Click Background and upload your new image.
- QA with the Preview tool.
- Click Start Experiment when ready.
Technical skills
None
Interpret results
- Win – Wes confirms that symmetric messaging works well for his visitors and that the outdoor adventure messaging resonates. He can use this information to design a more differentiated landing page or create a Personalization campaign to deliver more targeted messaging.
- Lose – Wes learns that his outdoor adventure imagery does not resonate with visitors who clicked his ad. He may want to investigate whether they are looking for a different type of adventure and if the holiday messaging is still more powerful for these visitors.
- Inconclusive – Wes can try a more radical change if he thinks there is a good opportunity.
Idea 7: Personalize based on cookies
Silvia knows that her travel site adds cookies that she can leverage to create a personalized experience. She hypothesizes that she can increase browsing behaviors and purchases if she customizes the site's homepage based on visitors' interests. For example, Silvia features a hero banner that highlights a destination related to the geographical area that a visitor frequently browses.
Hypothesis
Silvia thinks that visitors who browse certain locations are more likely to buy vacation packages in those areas. If she features destinations in that area in the hero banner for visitors who browse more than two pages, she can increase browsing behaviors and conversions.
Pages
The URL of the homepage
Audiences
Visitors who meet the audience condition for a unique cookie that Silvia can personalize, such as a cookie that tracks the geographical area of vacation destinations that a visitor has most recently browsed.
Metrics
- Clicks to the promoted hero banner (primary)
- Pageviews for the promoted destination (secondary)
- Clicks to Search (secondary)
- Pageviews for other destinations in the area (secondary)
Editor
Original:
Variation:
- Select the variation.
- Click the image to select the container.
- Click Background and upload your new image.
- QA with the Preview tool.
- Click Start Experiment when ready.
Technical skills
None
Interpret results
- Win – Silvia confirms that more personalized content improves browsing and purchase behaviors for her site. She can continue testing this concept by promoting vacation types instead of geographical areas. She can also start to deliver more granular Personalization campaigns, such as promoting amenities and services often purchased or browsed in certain geographical areas.
- Lose – The results do not support Sylvia's hypothesis. She may investigate why. Maybe her visitors prefer to start with a wide array of options instead of a targeted experience. Maybe many visitors are not primarily shopping by the geographical area—the price of the vacation is more important to them.
- Inconclusive – The experiment may not be bold enough. Silvia can try a lightbox or overlay for a more radical change.
More resources
Idea 8: Test promotion formats
Sarah's company offers a promotional price on a one-year subscription on its birthday each year. The offer generates a significant jump in new subscriptions, and research shows that even customers who see the offer without purchasing respond positively.
The offer is promoted through an email mailing list and a note in the header. This year, Sarah plans to experiment with making the anniversary promotion more visible. She decides to test whether a site-wide banner will increase subscriptions.
Hypothesis
Sarah knows that her company's anniversary sale increases the number of one-year subscriptions. If she promotes the promotion with a site-wide banner instead of the usual promotion in the header, more customers will see it and sign up for a discounted subscription package.
Pages
Every page on the site
You can use substring match if all your pages follow a pattern like yourdomain.com/.
Audiences
Everyone
Metrics
- Clicks to the banner link (primary)
- Pageviews for subscription details (secondary)
- Pageviews for the subscription form (secondary)
- Clicks to Submit for the form (secondary)
Editor
Original:
Variation:
Create (or ask your developer to help create) an Optimizely Experimentation extension: a template that helps you add custom features like carousels and banners without continuous developer support.
Note
Check out Optimizely Experimentation's library of pre-built, re-usable extensions.
- Go to Implementation > Extensions.
- Click Create New > Using JSON.
- Copy the JSON for the butterbar extension.
- Click Create Extension.
- Click More and enable the extension.
Next, add the extension to your variation.
- Go to your variation.
- Go to Create. When you scroll down, your butterbar extension displays under Create Options.
- Select the butterbar and edit it as desired. Remove the other promotion.
- Click Start Experiment when ready.
Technical skills
None (but you may need to know JSON to create other, similar extensions).
Interpret results
- Win – Sarah knows that this type of element creates more conversions than the header promotion they typically use. She can decide to use this for other promotional offers. She can also experiment with different messaging or offers for logged-in versus logged-out users (with cookie-based audience targeting).
- Lose – The butterbar variation is less successful at driving conversions. This is an opportunity for Sarah to find out why. She can look into her data to see if there are differences between segments, check whether the butterbar displays incorrectly for some visitors or browser types, or try a different design or message.
- Inconclusive – Sarah can try a bolder version of the butterbar. She may conclude that for this type of promotion, visitors respond similarly either way. She can use this insight for designing future header or banner options.
More resources
Idea 9: Optimize a form
Greg's company relies heavily on landing pages to generate leads for the sales team. The company uses the form inputs to validate high-quality leads and tailor marketing efforts. Greg has noticed that completion rates have fallen as more fields are added.
Greg believes that he can optimize the form by balancing the quality and quantity of information. He wants to experiment with the form to see if they can ask for enough information without adding extra friction to the funnel.
Hypothesis
Prospective customers experience the fields in the lead generation forms as friction. Removing extra fields to streamline the form can increase form completions by 5-10%.
Pages
The URL for the page with the form
Audiences
Everyone
Metrics
- Clicks to form submission (primary)
- Clicks to the navigation elements on the page (secondary)
- Clicks to other buttons on the form page (secondary)
Editor
Original:
Variation:
- Select the variation.
- Go to Changes > Create.
- Go to Element Change and select the field. Click Hidden to hide the feature without removing the field, or click Remove. Click Save.
- Repeat Step 3 to hide or remove the label.
- QA with the Preview tool.
- Click Start Experiment when ready.
Technical skills
None
Interpret results
- Win – The results support Greg's hypothesis that form fields are friction points for visitors. He may want to experiment further by removing more fields or using placeholder text instead of field labels for a cleaner look.
- Lose – The results do not support Greg's hypothesis. This tells him he has not identified the right problem or solution. He can choose to do more research to understand what assumptions he has made that this data contradicts. Maybe customers in his industry expect to provide a large amount of information, or something else is negatively affecting the form experience.
- Inconclusive – Greg may have tested too small a change; he can experiment with a more radical change by removing all but necessary fields. Greg could declutter the visual design and add or remove explanatory text.
Idea 10: Add social proof
Byron's team created a business intelligence report to help generate higher-quality test ideas. The report highlights the compelling impact of social proof. The site does not currently feature any social proof, but Byron knows the marketing team recently gathered a set of customer use cases and testimonials. He wants to experiment with adding social proof to a high-traffic spot to see if it helps customers convert.
Hypothesis
The payroll feature is relatively new and unproven in the market, but it is showcased and gets significant traffic. Adding social proof to the page shows visitors more evidence of success and can increase conversions on the Get Started button.
Pages
The URL of the feature page
Audiences
Everyone
Metrics
- Clicks to Get Started (primary)
- Clicks to the details page of the feature (secondary)
- Pageviews for the details page (secondary)
- Form submissions (secondary)
Editor
Original:
Variation:
- Select the variation.
- Go to Changes > Create.
- Select Insert HTML to add a testimonial.
- Insert your text in relation to another element on the page. Click Magnify, then click the selector to position the new text.
- Enter your text under HTML. Under HTML Placement, reposition the inserted text as you like.
- Click Save.
- QA with the Preview tool.
- Click Start Experiment when ready.
Technical skills
None
Interpret results
- Win – The results support Byron's hypothesis. He knows that social proof is compelling to his visitors and helps increase the conversion rate. He can build on this by adding a short video to boost the impact of the testimonial. If he knows the industry of some of his visitors, he can personalize the page with testimonials from businesses in the same one. He can also look for other opportunities on the site to experiment with social proof.
- Lose – The results do not validate his hypothesis. He can segment his results and check whether the variation performed better or worse for any particular groups of visitors.
- Inconclusive – The change Byron tested may be too small. He can run a low-effort test that makes a bigger impact. For example, he can add a picture to his testimonial to make it more visually compelling. He can find a more dramatic quote or feature the testimonial above the fold.