Feature Experimentation FAQs

Optimizely Feature Experimentation is feature management and A/B testing for product teams.

Pair Optimizely Feature Experimentation and Optimizely Web Experimentation products to empower your entire organization, from product to marketing teams, to experiment and build an optimized customer experience.

You can experiment anywhere in your technology stack with Optimizely's SDKs. With Optimizely Feature Experimentation, you can run experiments in various languages. If you are interested in testing in a different language, contact Optimizely.

You pass a user ID and feature flag key to Optimizely Feature Experimentation SDK, and this buckets people into the flag. So, if you are running multiple servers or programming languages with the same feature flag key, the SDK gives the same response (booleans and flag variables). The SDK gives the same answer based on the user information you have specified because Optimizely Feature Experimentation uses deterministic bucketing with MurmurHash3 to determine which experiments and variations should be active for a user.

The datafiledatafile is language-agnostic. You can use the same datafile from the same project across different SDKs and get consistent bucketing. This way, you can activate and track the exact same experiments across different SDKs and get consistent results.

Optimizely Feature Experimentation SDKs support bot detection as of version 2.1. Enable bot filtering directly from your account’s project settings. When the setting is enabled, events sent from web browsers have bot filtering applied automatically. Applying bot filtering to events sent from other environments requires configuration. For more context, read about bot filteringbot filtering.

You have to pass an ID, but it can be any hashed ID you’d like, as long as it is consistent with what you want to toggle. User IDs work best in most cases, but we have also seen customers use session IDs or even request-level IDs for experiments.

You are billed based on a monthly active user (MAU)-based model. For information, see Monthly active usersMonthly active users.

Optimizely Rollouts is free to all users and all companies. Optimizely Rollouts account users can see the basic Optimizely UI and can run one experiment at a time. If you would like to add more seats or MAUs, upgrade to the full functionality of Optimizely Feature Experimentation, or if you are interested in the client-side solution, Optimizely Web Experimentation, contact Optimizely to discuss your requirements.

Rollouts is the free plan for Optimizely Feature Experimentation intended for startups and teams looking to get started with feature flags and A/B tests. Rollouts come with a subset of Feature Experimentation features, including unlimited feature flags and controlled rollouts, and the ability to run one experiment at a time.

Teams ready to manage feature flags across more users and applications can upgrade to a paid plan to access additional collaborators and MAUs, tools for governance and security, technical email Support, and unlimited Targeting Rules, Projects, and Change History.

Paid Optimizely Feature Experimentation plans, which remove the one concurrent experiment limit, offer a more robust feature set along with best-in-class Support and services for organizations with more stakeholders invested in increasing experiment velocity and feature delivery.

In Optimizely Feature Experimentation, how many collaborators you can add depends on your plan. Contact your Customer Support Manager for information. Optimizely encourages you to invite the whole team to increase visibility, share results and insights, and deepen experimentation across your working group. Optimizely offers permissions by role and additional integrated program management tools, so your team can collaborate more effectively.

Optimizely Feature Experimentation can be ran anywhere in your text stack, enabling lightning-fast experimentation. Feature Experimentation offers both front-end and back-end SDKsfront-end and back-end SDKs along with Optimizely AgentOptimizely Agent that can be run as a microserver.

The server-side SDKs can address performance concerns associated with adding a JavaScript snippet that each user downloads in their browser (the approach of client-side solutions, including Optimizely Web Experimentation). The experiment code runs before the web page ever loads.

Optimizely Feature Experimentation SDKs are built to split traffic to experiments without making any network requests. Unlike some platforms, which call out to third-party servers for experiment decisions, all decisions are made in-memory using a cached copy of the datafile. The impact on latency is negligible.

In other words, Optimizely Feature Experimentation is faster because it does not make any blocking API requests to get decisions about which experiment variant to use; the device (your mobile phone or server) running the code makes that decision in under a millisecond. Optimizely Feature Experimentation does not slow down your end user's experience. However, there are still a few performance considerations to keep in mind as you scale your usage of:

Changes in your app depend on how frequently you refresh or poll your datafile. Updates are typically reflected in the datafile within a few seconds. However, this timing is not guaranteed and may vary depending on factors such as project size and infrastructure.

No network requests are required to evaluate feature flags and experiments, so there is no added latency. Fetch the datafile in advance, the SDK only needs access to the datafile contents. Download the datafile out-of-band from serving an individual request from your users.

After the datafile is downloaded, no network requests are needed to serve features to users. However, the SDK still sends network requests to track events. You can control when and how this happens by implementing a custom event dispatcher. For more information on event dispatchers, see the event dispatcher configuration topic for your SDKyour SDK.

During an app.optimizely.com interruption, you cannot log into Optimizely to update experiments or make other changes. During a service interruption, your application remains operational, but all flagsflags default to "off", so it is a best practice to save a fallback data file and defensively code defaults for flag variablesflag variables.

Optimizely Feature Experimentation uses a JSON datafileJSON datafile uploaded to a global CDN containing the data needed to deliver and track your applications' flags and experiments. In the rare hypothetical situation that the Optimizely CDN were to go down, the SDK would not be able to load any data on your feature flags or experiments, but it would not prevent your application from running. Instead, when you need to evaluate a feature flag or experiment, the Optimizely Feature Experimentation SDK returns the default values for its APIs. This would be roughly the equivalent of turning off all your flags and leaving the values of feature variables undefined.

It is most important to consider the critical connection between your application and the Optimizely CDN. To prepare for a rare service interruption, you can:

Additionally, you can view the Optimizely status page.

Optimizely does not have a concept of "requesting features." Optimizely does not make network calls to serve features or experiments to users, so there's zero latency. For information, see Does Optimizely Feature Experimentation make an API call to serve features to my users?.

Optimizely Feature Experimentation is built for product teams. Developers work with Feature Experimentation SDKs, but on many teams, product managers and analysts with less technical expertise configure features and experiments, manage rollouts, and analyze results in Optimizely Feature Experimentation.

Yes, you can use Optimizely Feature Experimentation using a local copy of your datafile and batch up events to send later when your app is back online.

Yes, you can to use Optimizely Feature Experimentation with a heavily cached site. This setup requires appropriate configuration at your CDN provider and some additional implementation in the backend. Contact your Customer Success Manager (CSM) to discuss options. You may also be interested in using Optimizely Feature Experimentation at the edgeOptimizely Feature Experimentation at the edge.

You probably do not need to modify your firewall to use Optimizely Feature Experimentation. If your firewall has egress (outbound traffic) restrictions, you need to allowlist cdn.optimizely.com and logx.optimizely.com.

You can use Optimizely as a service or include the Optimizely SDK in every service. For some best practices and special considerations when implementing Optimizely Feature Experimentation, see MicroservicesMicroservices.

Yes. Our SDKs need to be able to download our JSON payload (the "datafile"). The datafile contains all the metadata about your features that the SDK needs to operate (toggle feature flags, apply remote configuration ("feature variables"), and segment users). The SDKs do not need two-way communication with our servers.

It is possible to run the open-source Optimizely Agent as a microservice or to download the datafile from an authenticated endpoint, which minimizes the security risk. Some customers even relay this information to their own CDN for added security and control.

We have SDKs for all the major programming languages (including AndroidAndroid, C#C#, FlutterFlutter, GoGo, JavaJava, JavaScript SDK v6+JavaScript SDK v6+, JavaScript (Browser)JavaScript (Browser) (pre v6), JavaScript (Node)JavaScript (Node) (pre v6), PHPPHP, PythonPython, ReactReact, RubyRuby, and SwiftSwift), a REST APIREST API and great blog articles with examples for programming language. You can also run Optimizely AgentOptimizely Agent as a microservice (open source and free).

Yes. Some Optimizely Feature Experimentation users configure a Feature Experimentation environment for each customer. That way, you can use the same flag key everywhere and deliver progressively.

Other Optimizely Feature Experimentation users use key-value pairs to create audiences targeting specific customers to provision flags.

You must log into the Optimizely app and configure the feature flag using the same key. This also makes sure anyone logging into Optimizely knows what flags are where and why (you can label and describe them). You can also use the Optimizely REST API to change features.

If you prefer, you can also build your own UI or use it in combination with the Optimizely app. Feature Experimentation has a microservice, REST API, and command-line interface you can talk to.

With access to Optimizely export featuresaccess to Optimizely export features, you can export the raw events from AWS S3.

Optimizely provides guides for connecting with analytics platforms. For information, see Configure analytics platformsConfigure analytics platforms. For a full list of Optimizely partners, see Partners.

To measure which features people saw, run an experiment (as opposed to using a feature flag to deliver an "on" or "off" experience). To analyze your experiment results, you have the following options: