Closed-loop interface
This topic describes the closed-loop interface that is used for bi-directional handling of actions and data exchange between Optimizely Campaign and external systems. For example, using the closed-loop interface, you can create and send personalized newsletters automatically in Optimizely Campaign from an external campaign management system.
Setup
Contact customer support regarding setup of the closed-loop interface, and terms and conditions that apply when accessing this type of non-anonymized action-based data.
Tip
Because the response data consists of non-anonymized action-based data, you have to indemnify and hold Optimizely harmless in advance from any potential liabilities and third-party claims which may result from making this functionality available to you.
How it works
The closed-loop interface consists of two modules: the import module and the response data export.
To create and send mailings automatically, you must first create a source mailing in Optimizely Campaign and submit the mailing ID to your external system, for example, a campaign management system (CMS). In the CMS, you select the recipients and provide the recipient data.
The import module transfers the recipient data including the ID of the source mailing to Optimizely Campaign. After the import, you receive a status notification. In Optimizely Campaign, a campaign copy is created with the personalized content (for example, product recommendations) and the mailing is sent - either directly, or at a specified time. Optionally, you can resend the mailing manually if recipients have generated soft bounces, for example.
The response data export transfers the response data (opens, clicks, responses, unsubscribes) of the sent mailing back to your CMS. You can then process the response data further in the CMS or a third-party system, such as a data warehouse.
The import module and the response data export can be configured and used independently of each other.
Exchanging data
The data exchange is done using an SFTP server set up at Optimizely, and recipient data is transferred to that server. Response data is available for download from the same server.
Running transfers must have a temporary file name to avoid incomplete file exchange, see Data transfer troubleshooting. Naming conventions are described below.
Note
The data exchange does not respect strict separation of clients. All user data can be transferred through one SFTP account. You can set up client-specific SFTP accounts if required for security reasons. Alternatively, you can set up separate sub-directories in one client. In both cases, the recipient data import module must then commit one file per client/directory.
Transferring recipient data
Recipient data can be transferred at any time to the Optimizely Campaign SFTP server.
The name convention of the transferred files must be as follows:
YYYYMMDDHHMMSS_subscribers.csvThe recipient file is a CSV file with the following properties:
- Phrase and field lengths are variable within the limits of the recipient list.
- A semicolon (;) is used as separator.
- Line feed is used for line breaks.
- Quotation marks ("") are used as field boundaries. To designate quotation marks within a field, wrap them in quotation marks.
- The file does not contain declarations regarding columns and field lengths.
- Empty fields (NULL) remain empty in a data set. The number of field separators must be constant.
Recipient data is processed automatically every 10 minutes. The SFTP server of Optimizely Campaign provides log files for download within 30 days after creation.
Note
According to the specifications, fields of the main recipient list must be committed. Fields supposed to not contain any value are committed as empty columns. Existing fields and the field sequence are fixed. Do not change them without the consent of both parties.
closed-loop-interface.mdxUsing personalization and recommendations
It is possible to enhance the delivered recipient data with personalization, for example by adding product recommendations to the mailing.
One or more fields reserved for a product ID are added to the CSV file with the recipient data. Product information (such as title, description, image-URL) for recommended products is transferred in a second CSV file. Product information is inserted into the mailing using field functions and a placeholder. Assigning product IDs to recipients is done by the customer.
With web analysis software, recommendations are available via a CSV file. Other recommended products are assigned to each product. Product information is rendered into the mailing using a special field function, similar to the process described above. You do not need to assign products to a recipient, as this is automatically done by the web analysis software in accordance with the initial configuration.
Note
To add personalized content, you need a new template. Contact customer support.
closed-loop-interface.mdxCreating source mailings and sending mailings
As a basis for the campaign you are going to send, you create a source mailing in Smart Campaigns and submit the mailing ID to the external system. A so called master list serves as recipient list for the source mailing. Placeholders referencing a specific field of the recipient list are inserted into the mailing paragraphs. Using these placeholders you can individually assign and insert product data and other information into the mailing.
To send multiple newsletters, or send newsletters at the same time, you have to create multiple source mailings. The source mailing is never sent and must remain in status New or respectively Activation required. Do not activate the mailing, otherwise the copy cannot be created. Instead the system automatically creates a copy of it and sends that, so the source mailing can be reused.
Note
If you create a mailing using the REST API, you cannot use the mailing for the closed-loop interface. Create the source mailing only in Smart Campaigns.
Triggering the dispatch
When a file containing recipient data is stored in the corresponding directory on the SFTP server, the mailing is sent. Every 10 minutes, a Cron job checks this directory for new files. If a new file exists, it is imported into the recipient list of the corresponding client, and assigned to the source mailing using the parameter BROADMAIL_ID.
There are different ways to add content to a mailing:
- Directly from the file. The content is imported along with the recipient data in a CSV file. This method is only applicable for texts and hyperlinks.
- Referenced content. URLs in the imported CSV files are used to reference the content. When sending the mailing, the content is retrieved from the external server of the customer through the content interface. You can add texts, pictures, attachments, and hyperlinks types of content to a mailing.
When the data is imported, the mailing is automatically sent, and gets a unique ID.
You can send test emails using the closed-loop interface by storing test data instead of real data on the server. The CSV file must have the same structure as the file containing the real recipient data. Every time a test email is sent, a new mailing is created in Optimizely Campaign.
Note
Mailings are triggered only by storing a file on the server configured for the closed-loop interface. You cannot send mailings by using the Optimizely Campaign user interface.
You can customize the source mailing for different newsletters. Only change the source mailing after the previous mailing is sent. Additionally, ensure that the correct mailing ID of the source mailing is used in the recipient CSV file.
closed-loop-interface.mdxTroubleshooting data transfer
If a problem occurs transferring the data, a notification is sent to an email address configured beforehand. An error file is generated and the transfer stopped. During implementation you can configure the exact interface behavior in case of an error. When the implementation is completed, you can switch off this setting.
If an error occurs, the interface must be restarted in accordance with the error file. These are the cases:
- Data import failed.
- Error when mailing is started after data import.
Errors during data transfer are handled by customer support, since these can only be resolved manually.
closed-loop-interface.mdxMonitoring and sending status
After a recipient data import, the status of the process is sent via email and registered in a log file. This assures that actions within the interface are properly logged and documented in Optimizely Campaign.
Structure of the main recipient list
Column name | Data type | Example | Description |
|---|---|---|---|
Varchar(255) | Email address of the recipient | ||
broadmail_ID | Bigint/Long | 44018617811 | Mailing ID in Optimizely Campaign (broadmail_ID) |
WAVE_ID | Bigint/Long | 46623317811 | Identifies the selection/sending wave for unambiguous assignment of user actions. This parameter is optionally set by the external system when importing the recipient data. |
Salutation | Varchar(255) |
| Field from existing recipient list. |
First name | Varchar(255) |
| Field from existing recipient list. |
Last name | Varchar(255) |
| Field from existing recipient list. |
Note
The list configuration described above is an example. More fields can be added if required for personalization purposes. Contact customer support beforehand, and document them in the interface setup.
closed-loop-interface.mdxSpecifying response data
Response data is generated daily from the log files, and is available for download at a configured time via an SFTP server within Optimizely Campaign. The files contain the most recent data. The files containing the mailing data are available for download for 30 days after starting the mailing.
The response data consists of eight CSV files. Each of these can be read as a table. The response data is exported as String data type. You can also convert numeric values such as the mailing and client ID to the Long data type. See detailed type information for each file below.
Tip
The names used in the following documentation are examples. File names can be configured.
Sent mailings
YYYYMMDD_mailings.csv contains mailings sent in the referenced period.
Column name | Description | Example | Key |
|---|---|---|---|
id (mailingId) | Mailing ID | 51106527229 | Primary key |
compoundMasterId | Campaign ID in Smart Campaigns | 71206365463 |
|
mailingGroupId | Client ID (BROADMAIL_ID) | 44018617811 |
|
mailing.recipientLists | Recipient list ID | 58243723257 |
|
started | Start time of the mailing | 2021-01-01 12:07:55 |
|
finished | End time of the mailing | 2021-01-01 12:09:10 |
|
Sending Log
YYYYMMDD_mailingrecipients.csv contains recipient-related data. The sending log refers to the mailings contained in the file YYYYMMDD_mailings.csv.
Column name | Description | Example | Key |
|---|---|---|---|
id (mailingToUserId) | Encoded assignment of the recipient | 25951836752 | Primary key |
mailingId | Mailing ID | 51106527229 | Foreign key |
mailingGroupId | Client ID (BROADMAIL_ID) | 44018617811 |
|
userId | Email address of the recipient |
| |
created | Sending date and time | 2021-01-01 12:07:55 |
|
sendingStatus | Sending status | * sending - Mailing is sent | |
- sent - Mailing was sent
- could_not_be_sent - Mailing could not be sent | | | deliveryStatus | Delivery status | - OK - Mailing was delivered
- HARD - Hard bounce
- SOFT - Soft bounce | | | response.category | Response category | * unknown - The recipient sent a response to the mailing
- bounce - Soft bounce
- temporary_bounce - Soft bounce
- fatal_bounce - Hard bounce
- autoresponder - Autoresponder | |
Links
YYYYMMDD_links.csv contains links from the mailing. Data is only available for links with activated link tracking.
Column name | Description | Example | Key |
|---|---|---|---|
id (linkId) | Link ID | 44018617936 | Primary key |
mailingId | Mailing ID | 51106527229 | Foreign key |
mailingToUserId | Encoded assignment of the recipient | 25951836752 | Foreign key |
mailingGroupId | Client ID (BROADMAIL_ID) | 44018617811 |
|
name | Link name | Impressum |
|
url | Link target (URL) | http://www.example.com |
|
Clicks
YYYYMMDD_clicks.csv contains mailing clicks generated by recipients, within the referenced period. Clicked links/URLs can be found in the file YYYYMMDD_links.csv.
Column name | Description | Example | Key |
|---|---|---|---|
linkId | Link ID | 44018617936 | Foreign key |
mailingId | Mailing ID | 51106527229 | Foreign key |
mailingToUserId | Encoded assignment of the recipient | 25951836752 | Foreign key |
mailingGroupId | Client ID (BROADMAIL_ID) | 44018617811 |
|
userId | Email address of the recipient |
| |
created | Date and time of the click:::hint{style="info"} | | |
Note | | | |
There can be multiple clicks from one recipient within one second. ::: | 2021-01-01 12:07:55 | | | click.browser | Web browser | Firefox 64.1 | | | click.platform | Operating system | Windows 10 | | | click.device | Device type | Desktop | |
Opens
YYYYMMDD_opens.csv contains mailings opened by recipients in the referenced period.
Column name | Description | Example | Key |
|---|---|---|---|
linkId | Link ID | 44018617936 | Foreign key |
mailingId | Mailing ID | 51106527229 | Foreign key |
mailingToUserId | Encoded assignment of the recipient | 25951836752 | Foreign key |
mailingGroupId | Client ID (BROADMAIL_ID) | 44018617811 |
|
userId | Email address of the recipient |
| |
created | Date and time when the recipient opened the mailing:::hint{style="info"} | | |
Note | | | |
One recipient can open a mailing several times. In this case, each opening generates a singular data set. ::: | 2021-01-01 12:07:55 | | | open.browser | Web browser | Firefox 64.1 | | | open.platform | Operating system | Windows 10 | | | open.device | Device type | Desktop | |
Automatic opens
YYYYMMDD_automated_opens.csv contains automatic opens in the referenced period. See Apple Mail Privacy Protection.
Column name | Description | Example | Key |
|---|---|---|---|
mailingId | Mailing ID | 51106527229 | Foreign key |
mailingToUserId | Encoded assignment of the recipient | 25951836752 | Foreign key |
mailingGroupId | Client ID (BROADMAIL_ID) | 44018617811 |
|
userId | Email address of the recipient |
| |
created | Date and time of the generated open | 2021-01-01 12:07:55 |
|
open.browser | Web browser | Apple Mail Privacy Protection |
|
open.platform | Operating system | unknown |
|
Unsubscribes
YYYYMMDD_unsubscribes.csv contains unsubscribes by recipients in the referenced period.
Column name | Description | Example | Key |
|---|---|---|---|
mailingId | Mailing ID | 51106527229 | Foreign key |
mailingToUserId | Encoded assignment of the recipient | 25951836752 | Foreign key |
mailingGroupId | Client ID (BROADMAIL_ID) | 44018617811 |
|
userId | Email address of the recipient |
| |
created | Date and time of the unsubscribe | 2021-01-01 12:07:55 |
|
Responses
YYYYMMDD_responses.csv contains emails returned to the sender address from any user within the referenced period.
Column name | Description | Example | Key |
|---|---|---|---|
mailingId | Mailing ID | 51106527229 | Foreign key |
mailingToUserId | Encoded assignment of the recipient | 25951836752 | Foreign key |
mailingGroupId | Client ID (BROADMAIL_ID) | 44018617811 |
|
userId | Email address of the recipient |
| |
created | Sending date and time | 2021-01-01 12:07:55 |
|
category | Response category | * unknown - The recipient sent a response to the mailing | |
- bounce - Soft bounce
- temporary_bounce - Soft bounce
- fatal_bounce - Hard bounce
- autoresponder - Autoresponder | |