Use direct data for experimentation

Many organizations rely on direct data, or direct intelligence about their site experience, for ideation. Direct data can inform you about your visitors: who they are, what they are trying to do, and how they are trying to do it. You can then form data-driven hypotheses about the actions a visitor will likely take.

Below are some of the most common direct data sources. You do not need to consult all sources equally. Consider which can generate actionable insights to guide your experimentation. A few different data sources can give you a bigger picture of your visitors' site experience.

Use the insights to build a business intelligence report to support a strong experimentation program. 

Behavioral web analytics can help you understand where to optimize. Pair this data with other direct and indirect data sources to decide how to optimize for different opportunities on your site. For example, use pathing and funneling analyses to look for disproportionate drop-off rates, or monitor traffic patterns and page values (an estimate of a page's revenue generation) to find the most valuable content on your site. Finding that 20% of your site generates 80% of the value can help you focus your data collection and optimization.

Once you have identified bottlenecks, use qualitative data such as surveys and heatmaps for additional insight. Use indirect data such as reviews of competitor sites to note different design decisions that you might test.

See Use basic analytics reports or Use advanced analytics reports for more information on generating experiment ideas.

The most actionable source of quantitative data is often the results of your past experiments and campaigns. The difference between the success of two experiences can inform you about what your visitors want.

For example, if moving a button to a different place on the page results in a higher click-through rate, you have evidence that button placement matters to your visitors. You can then test if other, similar types of changes might matter.

Use web analytics, heatmaps, and survey results to provide context for the results of a past experiment. For example, a more prominent button encourages more click-throughs but results in more visitors exiting the site. Qualitative data can help you understand why and how to encourage visitors to stay.

Surveys, NPS scores, and the experiences of your support team can provide unique data about customer experience.

Optimizely uses NPS surveys to gather customer feedback on how the company can improve. Optimizely has used this feedback many times to bring products to market, such as Stats Engine.

See if your NPS scores have any actionable trends. Pair an NPS scoring system with different experiment variations to gather direct feedback from customers about each experience. These responses can explain why a page is underperforming and where you can optimize your site experience.

Heatmaps show where your visitors focus their attention and what part of the page is most important.

Landing pages are some of the most important places to optimize, particularly for B2B, lead generation, and consumer service businesses. Once you understand which parts get the most attention, redesign the page to highlight CTAs, headlines, or videos.

If your site has multiple landing pages, reducing the bounce rate across all can be difficult. Heatmaps show you trends; for example, if a certain element distracts visitors from your ultimate goal. You can also run a heatmap analysis on your competitor’s landing page or an admired third-party experience to compare.

User personas describe your customers using common themes. Interviews, surveys, analyses, and brainstorms help you create personas that inform your testing and personalization efforts.

User personas help frame campaigns and experiments. For example, use them to decide how to target headline copy to a C-suite executive, budget shopper, or social sharer. You do not have to develop full, formal profiles. Informal personas, such as customers who have not purchased in a long time, can be beneficial.

Combine user personas with web analytics to understand your customers' motivations. For example, certain shopping behaviors, like a cart with kitchenware and holiday decorations, can indicate a customer buying for a family. You can create a targeted message or promotional incentive for that persona.

Indirect data like competitor reviews can also impact your personas.

Usability testing shows common pitfalls in your site experience. Conduct manual tests with live sessions or use a platform like UserTesting.com to gather data asynchronously. You can answer questions such as:

Use survey results, competitor reviews, and web analytics to develop prompts for usability testing.

If your web analytics indicate poor performance on a landing page and a heat map shows that visitors do not engage with the CTA, design a prompt that asks customers to explain what they think the page is trying to communicate and whether it is compelling.

Some consumer decisions require multiple touchpoints before they reach a final decision. These touchpoints often occur offline as well as online. Smaller online touchpoints can be called micro-conversions.

Build a model to describe a customer's triggers and interactions during the decision-making process. This can help you optimize for the micro-conversions that lead a customer to complete the transaction.

Create hypotheses that focus on the most important micro-conversions in your customer decision model, including what triggers indicate success. Monitor how optimizing these micro-conversions affects your leading indicators.

Look for patterns among visitors who display the micro-conversion behaviors. Combine your customer decision model with your web analytics and personalization strategy to target CTAs and value propositions based on where a customer is in the decision-making process.

Once a visitor has seen your content, consider moving them deeper into the model; the visitor may be ready to provide an email address for additional content like technical specifications.

Monitor interactions between different metrics to understand how an improvement to one might detract from another. You can directly test these theories to optimize for aggregate profit.

For example, an online retailer might find that offering a promotional discount improves conversion rates at the expense of cart size and average order value (AOV). They might experiment with discount levels to model a price-elasticity curve. This model can help maximize total revenue based on tradeoffs between unit margins, purchase conversion rates, and AOV.

Examine site experiences, academic publications, and your competitor's sites to understand how they negotiate the tradeoffs. For example, research shows that business travelers are less likely to impact their travel itineraries for a lower price than leisure travelers. This industry-level insight provides context for how to optimize for different metrics.

Web analytics and heatmaps also help you design experiments that test different metrics. For example, the exit rate at different funnel stages can indicate when to introduce a promotional banner into your visitor’s experience.