A/B test
With an A/B test, you can test a mailing in different variants on a small portion of the recipient list before you start the actual dispatch. The variant that performs best in the criteria you defined is then optionally sent to the remaining recipients.
a-b-test.mdxThe variants (split mailings) usually differ only in few parameters from another. The most common variations include:
- Subject line. Test how different subject lines affect the opening rate.
- Header image, title, intro paragraph. Test various headers, titles and introductory paragraphs.
- Target group-specific testing. Test a mailing or a mailing variant to different target groups (for example, male and female recipients).
- Time of dispatch. Test the optimal time to send the master mailing.
A/B test in Smart Campaigns
To perform a A/B test in Smart Campaigns, do the following:
- Open the Optimizely Campaign menu and select Campaigns > Smart Campaigns.
- Click Create…
- From the left action pane, drag the Recipients node into the workspace and select one or more recipient lists.
- Drag an A/B Test node from the left action area and drop it onto the desktop.
- Connect the Recipients node to the A/B Test node.
- Drag a message node (Email or Mobile Push) onto the desktop. Create a new message or edit an existing message.
- Drag another message node of the same type onto the desktop. Select the message that you used for the first message node and change the content according to the criteria that you want to test.
- Add further message nodes if necessary and follow the description in point 7.
- Connect each message node to the A/B Test node.
- In the A/B Test node, optionally activate Activate automatic optimization and/or Send time optimization. For example, you can configure that the mailing with the best unique opening rate is automatically sent to the rest of the recipient list three days after the dispatch of the split mailings. For more information, see A/B Test node and Configuring send time optimization in the A/B Test node.
- In the Edit menu of the respective connection, you can set, among other things, how many recipients should receive the split mailing. For example, if you select 10% for two split mailings, the mailing will be sent to the remaining 80% of the recipient list after the A/B test has been completed. For more information, see A/B Test node.
- Click Save and Close. After the successful validation, you can activate the A/B test campaign.
a-b-test.mdxA/B test for transactional mails
With an A/B test, you can send different variants of a transactional mail and use the key figures to determine which variant performs better. To evaluate the mailing key figures, for example, you can use the Deep Analytics analysis tool.
To create an A/B test with transactional mails, do the following:
- Open the Optimizely Campaign menu and select Campaigns > Transactional Mails.
- Click Create….
- From the left action pane, drag the Recipients node into the workspace.
- In the Recipient Lists drop-down list in the context menu, select a transaction recipient list.
Note
Customer support has set up a separate transaction recipient list for you. Make sure you use this transaction recipient list, instead of a regular recipient list. To see if a recipient list is a transactional recipient list, check the column Transaction API.
- From the left sidebar, drag the A/B Test node into the workspace.
- Connect the Recipients node with the A/B Test node.
- From the left sidebar, drag at least two message nodes for the desired media type into the workspace. The marketing channel must be compatible with recipient list type. You can select the following marketing channels for A/B tests with transactional mails:
Marketing channel | Delivers | Required recipient list type |
|---|---|---|
Mail or multiple list | ||
SMS | SMS | SMS or multiple list |
Mobile Push | push message to a mobile device | Push or multiple list |
Web Push | push message that appears in the recipient's web browser | Push or multiple list |
- In the Name box in the context menu, enter a name for the transactional mail (for example Split 1).
- In the Description box, optionally enter a description for the transactional mail (for example Optimized subject line).
- Click Edit Content… and configure the message as described in Selecting the content type and Editing mailing content.
- Connect the A/B Test node with the message nodes.
- Click Properties on the outgoing connections.
- Specify to how many recipients the split mailing is to be sent.
- Percent. Enter the percentage of recipients who should receive the split mailing.
Note
The sum of the percentages in all outgoing connections must be 100 percent.
- Recipients. Enter the number of recipients who should receive the split mailing.
- No limit. Select this option if you do not want to limit the number of recipients.
Note
You can select this option only for one outgoing connection.
Tip
For example, if you enter 20% in the first split mailing and select No limit in the second split mailing, the second split mailing will be sent to the remaining 80% of recipients.
- Click Save and Close.
- To activate your transactional mails and start the sending process, select the mailing in the transactional mail overview and click Start.
Tip
To configure reports on each split mailing in Deep Analytics or to create target groups for segmentation, select the Transactional message mailing type.
A/B test in Mailings Classic
A split mailing in Mailings Classic is a variation of a regular mailing where a subset of recipients is "split off". You can edit every split mailing like a regular mailing and can also have a completely different look to the master mailing.
Note
A split mailing is associated with a master mailing, even if the master is moved to another folder.
To create split mailings in Mailings Classic, do the following:
- In the Mailings window, select the desired mailing – which is called the master mailing – and click Split. The Split Dispatch window opens.
- You have two options.
- Click Add split (master mailing) to create an exact copy of the master mailing with the same content. Edit this mailing by changing content elements, subject line, and so on.
- Click Add split (new mailing) to create a new mailing with no content. Add the desired content as described in Mailings-Classic.
In the Splits area, you can see the new split and existing splits of the selected master mailing. You can edit some parameters in the Split section of the Split Dispatch window, including:
- Name. This name is used internally. By default, the split receives the same name as the master mailing.
- Subject. Enter your desired variation to test different subject lines against one another.
- Maximum number of recipients. Specify the number of recipients whom the split should be sent (relative to master mailing recipients, in absolute numbers or a percentage).
- Target groups. Select up to three target groups for the split. See Target groups.
- Sending date. Set the dispatch date for each split. See Dispatch order.
- Click Save settings. The split is displayed in the mailing list. In the list, splits appear indented below the master mailing they belong to. In the Type column, they are designated as Split.
Recipients
To perform an A/B test, during which the best option between split A and B is selected and sent out as master mailing, splits are typically sent to a subset of total recipients. Each split can be sent to an absolute number of recipients, or a percentage of total recipients. If percentage is selected, it is based on the number of master mailing recipients. This principle also applies if a split uses a recipient list other than the master mailing.
For selecting a subset of recipients from a recipient list:
- Percentage. Enter a percentage in the Maximum recipients field. The number of recipients is based on the total number of master mailing recipients. Or, enter an absolute value. If the split contains the corresponding number of recipients, it is sent to exactly that number.
- Random. Recipients are randomly selected from the list. If you do not use this option, emails are sent to split recipients according to their order on the list.
In both instances, recipients who receive one split email will not be sent another. Similarly, recipients who are on several lists only receive one email. Optimizely Campaign monitors email addresses during dispatch to ensure that each recipient only receives one email (split or master mailing).
Recipient lists
In most cases, the same recipient lists are used for the master mailing and splits. However, for a split, you can use a recipient list other than master mailing. The master mailing recipient list is the basis for calculating the number of split recipients.
To use different recipient lists for the master and split mailing, in the Split Dispatch window, select off from the Automatically use the best split for the master mailing drop-down list.
If you are using different recipient lists in combination with recipient data-based target groups, be aware of potential conflicts between the different recipient lists. See Target groups.
Calculating recipients
The total number of recipients (master mailing and splits) is calculated every time before a mailing – including individual splits – is sent. The following illustration shows how subsets of recipients and the total number of recipients are calculated during dispatch.
Recipient calculation
- The symbol signifies that a subset of the total is being created.
- If a percentage is given for the number of split recipients, discrepancies between individual splits are possible, as cancellations and bounce overflows are taken into account for each case. This reduces the number of possible total recipients and the correspondingly calculated percentages.
- The number of possible recipients also considers the master mailing's target groups. If a target group with the recipient list of the master mailing contains fewer addresses than the same target group with the recipient list of the split, the number of recipients to be sent emails is the number calculated for the target group and recipient list of the master mailing.
Target groups
Target groups used in the master mailing are inherited by the split mailings. You can add target groups to split mailings by linking with the target groups of the master mailing using AND logic.
The number of matches of a target group in the recipient list of the master mailing furthermore determines the maximum possible number of recipients for this target group. If the recipient list of a split contains more matching recipients for this target group, the total number of recipients sent an email will not exceed the number previously calculated on the basis of the target group and recipient list of the master mailing.
Note
Master mailing vs. split target groups
Avoid target groups in the main and split mailings that exclude each other (for example, Male and Female), because no recipients will be selected for the split.
Note that the creation of target groups using recipient-based criteria can only apply when the corresponding criteria is also defined as a list box. Furthermore, if using a different recipient list for the split than for the master mailing, ensure that it incorporates the same data structure as the master mailing list – or at least that the list boxes for the master mailing target group can also be found in the split. The description of the corresponding list boxes must be an exact match, including capitalization.
Example
If a target group is used in the master mailing with the criteria Recipients with interest "Football", the recipient lists of both the master mailing and the split must contain a common corresponding field, such as Interests.
Dispatch
To send a split mailing in the Split window, click Start Split or select the check box below Mailing schedule and specify a dispatch date using the calendar.
a-b-test.mdxDispatch order
Set the order of splits in the lower section of the Split Mailing window via arrow keys to the right of each split. Splits associated with a master mailing are part of a hierarchy, which is determined by the order of the splits. This split hierarchy is referenced when calculating recipients of splits and the master mailing.
Note
Split hierarchy vs. sending order
When individual splits are manually started, the split hierarchy applies, without regard for dispatch order. For example, if the last split in a list is sent first, the recipients for the split and master mailings are calculated on the basis of the overall split hierarchy.
Under certain circumstances, this may lead to a split being sent to fewer or even no recipients despite having started first. This is because the splits higher up on the list are allocated the complete number of recipients, and no recipients remain for the subordinate split. For more information, see Split mailing examples.
If the split dispatch is started automatically, the split mailings are sent in the order of the split hierarchy.
Optimizing and sending the master mailing
An A/B test lets you analyze each split's performance and send the best performing variation as a master mailing.
Optimizely Campaign also offers an automatic optimization. To activate it, select one of the following criteria from the Automatically use the best split for the master mailing drop-down list. This criterion determines which split is chosen for the master mailing.
Note
Which criterion you use depends on mailing type, the quality of recipient data, and your marketing goals.
- Best unique open rate. Split most opened by recipients (percentage).
- Best unique click rate. Split most clicked by recipients.
- Best effective click rate. Split with the best unique opens/unique clicks rate.
- Lowest unique unsubscribe rate. Split that created the fewest unsubscribes.
- Highest turnover. Split that created the highest revenue. This criterion only delivers valid results if post-click tracking was configured for your client; otherwise splits have the result 0.
- Lowest bounce rate. Split that created the fewest bounces.
There are three options to start the master mailing.
- Save mailing and start it manually later. Start the master mailing manually by clearing the check boxes and selecting the default option when editing the mailing in the Options tab.
- Sending date for the master mailing. Set a sending date for the master mailing and select the check box to activate this option. Use this option if you usually send your newsletter at a certain day and time.
- Sending delay for the master mailing after the last split mailing has been sent. Set a delay between the sending of the last split mailing and the master mailing. Select the check box to activate this option.
Tip
Use this option to get reliable opens and click rates based on the split mailings, because not all recipients open and click the mailing immediately upon receiving it.
a-b-test.mdxSplit mailing examples
The following examples to help you understand the split module by showing recipient calculations and their results. They illustrate the hierarchies and dependencies of the master mailing and splits.
In all examples, it is assumed that recipients are valid (that is, no one is blocked, has canceled, or exceeded the bounce limit).
a-b-test.mdxExample 1
The master mailing and splits are sent to the same recipient list. A limit on the recipient number is set in the master mailing. The splits comprise A/B testing (split A against split B) with different target groups. The recipient list contains 100 recipients (40 male, 40 female, 20 unspecified).
Mailing | Recipient list | Max. number of recipients | Target group | Sent emails |
|---|---|---|---|---|
Master mailing | A | 28% (= 28 recipients) | — | 0 |
Split 1 | A | — | male | 28 |
Split 2 | A | — | female | 0 |
Result: The total number of recipients, as defined in the master mailing, was reached during the dispatch of split 1. No emails are sent to Split 2 and the master mailing.
a-b-test.mdxExample 2
Master mailing and splits are sent to different recipient lists. A limit for the total number of recipients is defined in the master mailing. Recipient list A (master mailing) contains 200 recipients (60 male, 60 female, 80 unspecified). Recipient list B (splits) contains 100 recipients (40 male, 40 female, 20 unspecified).
Mailing | Recipient list | Max. number of recipients | Target group | Sent emails |
|---|---|---|---|---|
Master mailing | A | 28% (= 56 recipients) | - | 0 |
Split 1 | B | — | male | 40 |
Split 2 | B | — | female | 16 |
Result: The total number of recipients is determined by recipient list A and the limit of recipients defined in the master mailing. Split 1, only sent to male recipients with no sending limit, is sent to male recipients in list A. Split 2 is subsequently sent to 16 female recipients from the same list. At this point, the total number of recipients limit is reached.
Example 3
The master mailing and splits are sent to the same recipient list. Recipient list A contains 100 recipients (20 male, 20 female, 60 unspecified). Three splits are created.
- Split 1 has the target group "male"
- Split 2 "female"
- Split 3 has the target group "male OR female" (so gender must be specified).
The master mailing does not define a target group. Split 3 is to be sent first.
Mailing | Recipient list | Max. number of recipients | Target group | Sent emails |
|---|---|---|---|---|
Master mailing | A | — | — | 60 |
Split 1 | A | 10% | female | 10 |
Split 2 | A | 10% | male | 10 |
Split 3 | A | 10% | unspecified | 0 |
Result: While the first two target groups are subsets of the master mailing target group, the target group for split 3 and the master mailing exclude one another. To select recipients who did not specify gender, the target group of the master mailing should be "males OR females OR blank field".