Implementation checklist

Before implementing Optimizely Feature Experimentation in a production environment, review the configuration details and best practices checklist to simplify implementation.

The following diagrams demonstrate how you, your users, and Optimizely Feature Experimentation interact. Understanding these interactions is crucial for implementing Feature Experimentation effectively. Click on any diagram to enlarge it.

The overall architectural diagram provides a detailed view of Feature Experimentation's components and their interactions, illustrating how data flows between key entities. The diagram contains the following three main sections: Your implementation, Optimizely UI, and the Optimizely servers. It shows how these sections work together to support feature flagging and experimentation. It highlights interactions between components using Crow's foot relationship notation, a standard entity-relationship (ER) modeling technique that visually represents the types of relationships, such as 'one' or 'many,' between different parts of the application and event processing. See the key following the diagram.

The system context diagram shows how your application interacts with Optimizely and third-party analytics systems. It highlights how applications send event data to Optimizely, how decision data flows back, and how integrations with analytics platforms provide experimentation insights.

The SDK and app container diagram shows how the SDKs integrate within your application to evaluate feature flags and experiments. It covers feature configuration, decision-making, and event tracking. Network requests use JSON over HTTPS, and internal data exchanges use JSON.

The datafile is a JSON representation of the OptimizelyConfig. It contains the data needed to deliver and track your experiments and flag deliveries for an environment in your Optimizely Feature Experimentation project.

You have the following options for syncing the datafile between your Feature Experimentation project and your application:

Other important considerations for datafile management include the following:

The Feature Experimentation SDKs are configurable to meet the needs of any production environment. However, adequate scaling may require overriding some default behavior to better align with your application's needs.

Each SDK includes a logger framework that does nothing by default. You should create a custom logger and pass it to the Optimizely client. For example, you can write logs to an internal logging service or a third-party vendor.

For information, see the documentation for the logger and the SimpleLogger reference implementation.

In a production environment, you must handle errors consistently across the application. The Optimizely Feature Experimentation SDKs let you provide a custom error handler to catch configuration issues like an unknown experiment key or unknown event key. This handler should let the application fail gracefully, maintaining a normal user experience. It should also send alerts to an external service, such as Sentry, to alert the team of an issue.

Building a user profile service (UPS) helps maintain consistent variation assignments between users when test configuration settings change.

The Optimizely Feature Experimentation SDKs bucket users using a deterministic hashing function. This means that the user evaluates to the same variation as long as the datafile and user ID are consistent. However, when test configuration settings are updated (such as adding a new variation or changing traffic allocation), the user's variation may change and alter the user experience. See How bucketing works in Optimizely Feature Experimentation.

A UPS solves this by persisting information about the user in a datastore. At a minimum, your UPS should create a mapping of user ID to variation assignment. Implementing a UPS requires exposing a lookup and save function that returns or persists a user profile dictionary. This service relies on consistent user IDs across all use cases and sessions.

For example, using Redis or Cassandra for the cache, you can store user profiles in a key-value pair mapping. You can use a hashed email address mapping to a variation assignment. To keep sticky bucketing for six hours, set a time to live (TTL) on each record. As Optimizely buckets each user, the UPS interfaces with this cache and makes reads and writes to check assignments before bucketing normally.

Many developers use wrappers to encapsulate SDK functionality and simplify maintenance. You can do this for the previous configuration options.

Optimizely Feature Experimentation's environments lets you confirm behavior and run tests in isolated environments, like development or staging. This makes it easier to deploy tests in production safely. Environments are customizable and should mimic your team's workflow. Most customers use two environments, development and production. This lets engineering and QA teams inspect tests safely in an isolated setting while site visitors are exposed to tests running in the production environment.

View production as your real-world workload. A staging environment should mimic all aspects of production so you can test before deployment. In these environments, all aspects of the SDK, including the dispatcher and logger, should be production-grade. In local environments like test or development, it is okay to use the out-of-the-box implementations instead.

Environments are kept separate and isolated from each other with their own datafiles. For added security, Optimizely Feature Experimentation lets you create secure environments, which require authentication for datafile requests. The Feature Experimentation server-side SDKs support initialization with these authenticated datafiles. You should use these secure environments only in projects that exclusively use server-side SDKs and implementation. If you fetch the datafiles in a client-side environment, they may become accessible to end-users.

Optimizely Feature Experimentation runs on Google Cloud Platform (GCP) infrastructure. All server-side timestamps use Coordinated Universal Time (UTC). Infrastructure clocks sync using the Network Time Protocol (NTP) via Google's time service across all servers. This ensures audit logs, security events, experiment data, and forensic analysis records maintain consistent and accurate timestamps across the platform.

Optimizely syncs all infrastructure clocks to an accurate and authoritative time source using the following approach:

Feature Experimentation SDKs generate event timestamps locally. For example, impression and conversion events are timestamped when the SDK creates them, not when Optimizely servers receive them. Accurate system clocks are important because clock drift on your servers can cause event timestamps to diverge from Optimizely server-side timestamps, affecting experiment result accuracy and event correlation.

Batched events dispatch on a flush interval (for example, every one second in browsers or every 30 seconds for server-side SDKs). If local clocks drift, the gap between event creation and server receipt timestamps can complicate debugging and compliance auditing. See event batching for more information.

To maintain timestamp consistency between your systems and Feature Experimentation, sync your local clocks with a reliable NTP time source:

Vulnerability management responsibilities are shared:

User IDs identify the unique users in your tests. In a production setting, it is especially important to carefully choose the type of user ID and set a broader strategy for maintaining consistent IDs across channels. See Handle user IDs.

Attributes let you target users based on specific properties. In Feature Experimentation, you can define which attributes should be included in a test. Then, in the code itself, you can pass an attribute dictionary on a per-user basis to the SDK, which will determine which variation a user sees.

Build custom integrations with Optimizely Feature Experimentation using a notification listener. Use notification listeners to programmatically observe and act on various events within the SDK and enable integrations by passing data to external services.

The following are a few examples:

Before you go live with your experiment, review the following documentation:

If you have questions, contact Support. If you think you have found a bug, file an issue in the SDK’s GitHub repository.