Optimizely A/B testing (legacy)
Optimizely A/B testing (legacy) lets editors create variations for several page elements (blocks, images, content, buttons, and form fields), and then compare which variation performs best. It measures the number of conversions from the original (control) against the variation (challenger). The one generating the most conversions during the test period is promoted to the design for that page. A/B testing has several predefined conversion goals; you can also create custom conversion goals.
See also Experimentation for information on the Optimizely delivery and experimentation platform for websites, mobile apps, smart devices, and back-end code. With this, you can A/B test everything from search results and promotions to recommendations and payment options. You can also A/B test user interface variations, to optimize and personalize website messaging.
Requirements
- No additional license fee.
- An Optimizely Content Management System (CMS) installation.
- See App platform compatibilityApp platform compatibility for package and version information.
Install
Documentation
- Optimizely A/B testing (legacy) - user guideOptimizely A/B testing (legacy) - user guide
- Blog: The A/B testing addon for Optimizely CMS is now open-source by Kevin Shea
Technical limitation
The underlying tool used by A/B testing to render preview images of the changes under test has the following technical limitations:
- Media queries defined on link elements are ignored
- Media queries defined on import rules are ignored
This limitation results in the styles, which would ordinarily be restricted to particular types of media, being applied to screen media and therefore appearing in the preview images that are generated.
However, the functionality respects media queries defined directly within the CSS. So, a workaround for these situations is to wrap the existing contents of the linked CSS source file in a print media query.
For example, imagine that a file named print.css is linked to a page with a media attribute for print.
<link src="print.css" rel="stylesheet" media="print" />And imagine that print.css currently contains the following:
html, body { margin: 0; }
img { display: none; }Applying such a workaround would require you to wrap the existing statements in a media query:
@media print {
html, body { margin: 0; }
img { display: none; }
}The attribute can also remain on the element if that is preferable for other technical concerns.
Key performance indicator
A key performance indicator (KPI) in Optimizely records when a visitor on a website performs a desired action, such as clicking on an ad or a button, or completing a sale. The functionality is used in A/B testing of content.
You can create custom KPIs, using goal objects in the KPI framework, to use in Optimizely A/B testing or in any other package that relies on creating instance-based goal objects.
To create a custom KPI, implement the IKpi interface located in the EPiServer.Marketing.KPI.Manager.DataClass namespace.
Cookie usage in A/B testing
The AB testing package has one cookie per test the user visits in the format of:
EPI-MAR-<Content GUID>
"Content GUID" is the Optimizely GUID for the content under test. The visitor's state against the running test is stored inside the cookie. That is, the version the visitor should see if returning to a test item while the test is still running if they have yet to view the content in question, and the various goals the visitor has converted on or has yet to convert on.
- EPI-MAR-{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx:xx} – Analytical or performance functionality. Records a visitor's interaction with a website optimization test, to ensure a visitor has a consistent experience. Persistent (variable according to the test). Typically optimization tests are short-lived (one week). The cookie is removed after the test is completed.
See CookiesCookies for information about other cookies in the Optimizely platform.
IKpi methods
- Guid Id – The Id of the KPI instance. Used to identify between multiple instances of the same type of KPI.
- string FriendlyName – Display name of the KPI. Used for any calling package to identify the KPI type in a user-friendly manner.
- string Description – Describes what the KPI does so that it can be displayed to the user.
- string UiMarkup – The HTML form inputs required to create an instance of the KPI. The Validate method uses this to store relevant information the KPI needs to evaluate properly. Inputs require a name attribute to retrieve the value and pass it to the Validate method.
- string UiReadOnlyMarkup – The HTML fragment used for displaying the user-generated input that created the instance of the KPI.
- void Validate(Dictionary<string, string> kpiData) – Reads a dictionary of input generated from the UiMarkup to create a KPI instance. Keys in the kpiData Dictionary match the name property of the inputs in the UiMarkup. Stored values must be retrieved later in the Evaluate method as class properties. Generate a KpiValidationException when the data is not in a valid format so that the calling method can handle validation errors.
- event EventHandler EvaluateProxyEvent – Called by packages using this KPI, the EvaluateProxyEvent attaches the passed-in event proxy to the .Net event the KPI cares about. When the proxy is attached, it calls Evaluate with the proper sender and event arguments so the KPI can determine if a conversion occurred.
- IKpiResult Evaluate(object sender, EventArgs e) – Determines if a conversion occurred when the proper .Net event is executed. The passed-in EventArgs should be cast to the expected EventArgs type to check the Kpi’s instance data against the event-specific data.
- DateTime CreatedDate – Determines when the KPI instance was created.
- DateTime ModifiedDate – Determines when the KPI instance is modified.
- ResultComparison – Indicates how the KPI Evaluation result object should be interpreted when deciding if a result is better than another instance of the result. Used specifically for indicating whether a greater or lesser result is desired.
- void Initialize – Called by external packages when the KPI needs to set up dependencies that occur outside of construction. Designed to do any extra set up for the KPI, such as listening for setup events outside of the normal Evaluate event handler.
- void Uninitialize – Called by external packages when the KPI needs to clean up dependencies set up during the Initialize method.
- string KpiResultType – Informs external packages what type of KPI result evaluation will return.
IClientKpi
IClientKpi is an interface for marking a custom KPI that should be run on the client browser to convert an A/B test. It consists of only one method for retrieving the client JavaScript that must be presented in the browser to indicate when a conversion occurs.
- string ClientEvaluationScript – Returns a JavaScript function to be executed in a visitor’s browser to indicate when a conversion has occurred. This function’s first parameter should be a success callback function to execute when a conversion has occurred.
📘 Note The function returned by this method should be immediately invoked when loaded in the target page.
Example script to return:
function (success) {
if (mySuccessConditionWasMet()) {
success(); // Invoke the passed in success callback}
}Serialization
Calling packages such as A/B testing package rely on Microsoft’s System.Runtime.Serialization’s attributes to inform the user interface of a KPI type’s information. The [DataContract] attribute should be on the class implementing the IKpi interface and the [DataMember] attribute on fields that should be sent to the user interface.
Example: Custom KPI multiple target page conversion
using System;
using System.Collections.Generic;
using System.Runtime.Serialization;
using EPiServer.Marketing.KPI.Manager.DataClass;
using EPiServer.Marketing.KPI.Results;
using EPiServer.ServiceLocation;
using EPiServer;
using EPiServer.Core;
using EPiServer.Marketing.KPI.Exceptions;
using EPiServer.Marketing.KPI.Manager.DataClass.Enums;
namespace Devsite.KPI {
[DataContract]
public class DemoKPI: IKpi {
private string _description = "Conversion goal is activated when a user lands on one of the pages specified for the test.";
private string _friendlyName = "Demo Landing Section Goal";
private string _uiMarkup = "<div><input name=\"PageIds\" type = \"text\" placeholder = \"Pages\" /><span> Comma separated list of page ID's</span></ div>";
private string _readonlyMarkup = "<div>You chose some pages that I am not going to display, but could</div>";
private IServiceLocator _serviceLocator;
public DemoKPI() {
_serviceLocator = ServiceLocator.Current;
}
[DataMember]
public Guid Id {
get;
set;
}
[DataMember]
public List < Guid > PageGuids;
[DataMember]
public DateTime CreatedDate {
get;
set;
}
[DataMember]
public DateTime ModifiedDate {
get;
set;
}
[DataMember]
public string Description {
get {
return _description;
}
}
[DataMember]
public string FriendlyName {
get {
return _friendlyName;
}
}
[DataMember]
public string UiMarkup {
get {
return _uiMarkup;
}
}
[DataMember]
public string UiReadOnlyMarkup {
get {
return _readonlyMarkup;
}
}
[DataMember]
public ResultComparison ResultComparison {
get {
return ResultComparison.Greater;
}
}
[DataMember]
public string KpiResultType {
get {
return typeof (KpiConversionResult).Name.ToString();
}
}
private EventHandler<ContentEventArgs>_eventHander;
/// <summary>
/// EventHandler that tells the code using this KPI what CMS event this KPI wants to attach to in order to properly evaluate
/// </summary>
public event EventHandler EvaluateProxyEvent {
add {
_eventHander = new EventHandler<ContentEventArgs>(value);
var service = _serviceLocator.GetInstance<IContentEvents>();
service.LoadedContent += _eventHander;
}
remove {
var service = _serviceLocator.GetInstance <IContentEvents>();
service.LoadedContent -= _eventHander;
}
}
/// <summary>
/// Method that should get called when the CMS event we attach to in EvaluateProxyEvent that checks to see if a particular condition is met
/// </summary>
/// <param name="sender">The caller of the event</param>
/// <param name="e">Generic event arguments that should be cast to the specific event arguments defined by the event attached to in EvaluateProxyEvent</param>
/// <returns></returns>
public IKpiResult Evaluate(object sender, EventArgs e) {
var retval = false;
var ea = e as ContentEventArgs;
if (ea != null && PageGuids != null) {
retval = PageGuids.Contains(ea.Content.ContentGuid);
}
return new KpiConversionResult() {
KpiId = Id, HasConverted = retval
};
}
/// <summary>
/// Validates that the passed in data is able to be used to create an instance of the KPI
/// </summary>
/// <param name="kpiData">dictionionary of data used to validate and create instances of the KPI for use in other classes (like AB Tests)</param>
public void Validate(Dictionary<string, string> kpiData) {
var ids = kpiData["PageIds"];
if (!string.IsNullOrEmpty(ids)) {
PageGuids = GetPageGuidsFromContentIds(ids);
} else {
throw new KpiValidationException("Empty Page Id's");
}
}
private List <Guid>GetPageGuidsFromContentIds(string ids) {
var retList = new List <Guid>();
var aContentLoader = _serviceLocator.GetInstance<IContentLoader>();
foreach(var id in ids.Split(',')) {
int aId;
if (int.TryParse(id, out aId)) {
var aContentReference = new ContentReference(aId);
var aPage = aContentLoader.Get<IContent>(aContentReference);
retList.Add(aPage.ContentGuid);
} else {
throw new KpiValidationException("unable to parse the Ids - they should be a comma separated list of integers");
}
}
return retList;
}
public void Initialize() {
// not needed in this example
// a hook to attach to external events other than the evaluate event to set up data in the KPI instance here
// particularly useful for KPI's that rely on multiple user actions to happen before evaluate should return a converted result
}
public void Uninitialize() {
// not needed in this example
// a hook to clean up any objects set up during the intialize event
}
}
}