Run A/B tests

17 min

If you are new to experimentation, you can get a lot done with a simple ON or OFF A/B test. This configuration has one flag with two variations:

One "flag_on" variation.

One "flag_off" variation.

Restrictions

If you intend to have multiple experiments and flag deliveries (targeted deliveries) in your flag, your experiment must always be the first rule in your ruleset for a flag.

Configuration overview

To configure a basic A/B test, complete the following:

(Prerequisite) Create a flagCreate a flag.

(Prerequisite) Handle user IDsHandle user IDs.

Create and configure an A/B Test ruleCreate and configure an A/B Test rule in the Optimizely app.

Integrate the example decide codeIntegrate the example decide code that the Optimizely app generates with your application.

Test your experiment in a non-production environment. See QA and troubleshootQA and troubleshoot.

Discard any QA user events and enable your experiment in a production environment.

Create an experiment

With the Feature Experimentation REST API

Use the Update the Ruleset for a Flag in an EnvironmentUpdate the Ruleset for a Flag in an Environment endpoint to add the rule with the rule's type set to a/b.

Important For PATCH requests, first `GET the current ruleset`GET the current ruleset, merge only the fields you need to change, and then send the PATCH. This helps prevent overwriting existing settings you did not intend to modify.

Endpoint – PATCH https://api.optimizely.com/flags/v1/projects/PROJECT_ID/flags/FLAG_KEY/environments/ENVIRONMENT_KEY/ruleset

Authentication – Include your API key in the Authorization header as a Bearer token. See Generate tokens and use the REST APIsGenerate tokens and use the REST APIs.

Request example

curl --request PATCH \
     --url https://api.optimizely.com/flags/v1/projects/PROJECT_ID/flags/FLAG_KEY/environments/ENVIRONMENT_KEY/ruleset \
     --header 'accept: application/json' \
     --header 'authorization: Bearer TOKEN' \
     --header 'content-type: application/json-patch+json' \
     --data '
[
  {
    "op": "add",
    "value": {
            "key": "stock_message_test",
            "name": "Stock Message Test",
            "type": "a/b",
            "distribution_mode": "manual",
            "percentage_included": 10000,
            "metrics": [
                {
                    "aggregator": "unique",
                    "display_title": "add_to_cart",
                    "event_id": EVENT_ID,
                    "event_type": "custom",
                    "scope": "visitor",
                    "winning_direction": "increasing"
                }
            ],
            "variations": {
                "off": {
                    "key": "off",
                    "name": "Off",
                    "percentage_included": 5000
                },
                "on_hide_amounts": {
                    "key": "on_hide_amounts",
                    "name": "On Hide Amounts",
                    "percentage_included": 5000
                }
            },
            "audience_conditions": []
        },
    "path": "/rules/stock_message_test"
  }
]
'

Response example

{
  "url": "/projects/PROJECT_ID/flags/a_b_test_flag/environments/development/ruleset",
  "update_url": "/projects/PROJECT_ID/flags/a_b_test_flag/environments/development/ruleset",
  "enable_url": "/projects/PROJECT_ID/flags/a_b_test_flag/environments/development/ruleset/enabled",
  "rules": {
    "stock_message_test": {
      "key": "stock_message_test",
      "name": "Stock Message Test",
      "description": "",
      "variations": {
        "off": {
          "key": "off",
          "name": "Off",
          "percentage_included": 5000,
          "variation_id": 1500962,
          "id": 6669660
        },
        "on": {
          "key": "on",
          "name": "On",
          "percentage_included": 5000,
          "variation_id": 1500963,
          "id": 6669661
        }
      },
      "baseline_variation_id": null,
      "type": "a/b",
      "distribution_mode": "manual",
      "id": 1598853,
      "urn": "rules.flags.optimizely.com::1598853",
      "archived": false,
      "enabled": false,
      "created_time": "2025-09-12T20:39:20.565416Z",
      "updated_time": "2025-09-12T20:39:20.554081Z",
      "audience_conditions": [],
      "audience_ids": [],
      "percentage_included": 10000,
      "metrics": [
        {
          "event_id": 5312230607028224,
          "event_type": "custom",
          "scope": "visitor",
          "aggregator": "unique",
          "winning_direction": "increasing",
          "display_title": "add_to_cart",
          "alias": "dca1a5a6b8ea8c1beca2648e739e306e5d6a9399b041cbee72594ce75ef84d0b",
          "id": "46135e38-8e67-4656-826b-b3343d564f03",
          "project_id": PROJECT_ID,
          "visibility": "experiment",
          "updated_at": "2025-09-12T20:39:21.758625965Z",
          "account_id": ACCOUNT_ID,
          "event_scoped": false
        }
      ],
      "allow_list": {},
      "group_rule": {
        "group_id": 0,
        "traffic_allocation": 0
      },
      "group_remaining_traffic_allocation": 100,
      "layer_id": 9300001985314,
      "layer_experiment_id": 9300002731026,
      "status": "draft"
    }
  },
  "rule_priorities": [
    "stock_message_test"
  ],
  "id": 1881472,
  "urn": "rulesets.flags.optimizely.com::1881472",
  "archived": false,
  "enabled": false,
  "updated_time": "2025-09-12T20:39:20.642465Z",
  "flag_key": "a_b_test_flag",
  "environment_key": "development",
  "environment_name": "Development",
  "environment_id": 761744660106841,
  "default_variation_key": "off",
  "default_variation_name": "Off",
  "revision": 2,
  "status": "draft",
  "role": "admin"
}

See the Update the Ruleset for a Flag in an EnvironmentUpdate the Ruleset for a Flag in an Environment endpoint reference documentation for information on creating an A/B test with the API. You can also follow the REST API cookbookREST API cookbook and see step six to create an A/B teststep six to create an A/B test.

With the Feature Experimentation UI

See Run A/B tests in Feature ExperimentationRun A/B tests in Feature Experimentation in the user documentation.

Implement the experiment using the decide method

Flag is implemented in your code

If you have already implemented the flag using a Decide method, you do not need to take further action (the SDKs are designed so you can reuse the exact flag implementation for different flag rules).

Flag is not implemented in your code

If the flag is not implemented yet, copy the sample flag integration code into your application code and edit it so that your feature code runs or does not run based on the output of the decision received from Optimizely.

Use the Decide method to enable or disable the flag for a user:

// Decide if user sees a feature flag variation
user := optimizely.CreateUserContext("user123", map[string]interface{}{"logged_in": true})
decision := user.Decide("flag_1", nil)
enabled := decision.Enabled

// Decide if user sees a feature flag variation
var user = optimizely.CreateUserContext("user123", new UserAttributes { { "logged_in", true } });
var decision = user.Decide("flag_1");
var enabled = decision.Enabled;

// Decide if user sees a feature flag variation
var user = await flutterSDK.createUserContext(userId: "user123");
var decideResponse = await user!.decide("flag_1");
var enabled = decideResponse.decision!.enabled;

// Decide if user sees a feature flag variation
OptimizelyUserContext user = optimizely.createUserContext("user123", new HashMap<String, Object>() { { put("logged_in", true); } });
OptimizelyDecision decision = user.decide("flag_1");
Boolean enabled = decision.getEnabled();

// Decide if user sees a feature flag variation
const user = optimizely.createUserContext('user123', { logged_in: true });
const decision = user.decide('flag_1');
const enabled = decision.enabled;

// Decide if user sees a feature flag variation
$user = $optimizely->createUserContext('user123', ['logged_in' => true]);
$decision = $user->decide('flag_1');
$enabled = $decision->getEnabled();

# Decide if user sees a feature flag variation
user = optimizely.create_user_context("user123", {"logged_in": True})
decision = user.decide("flag_1")
enabled = decision.enabled

// Decide if user sees a feature flag variation
var [decision] = useDecision('flag_1', null, { overrideUserAttributes: { logged_in: true }});
var enabled = decision.enabled;

# Decide if user sees a feature flag variation
user = optimizely_client.create_user_context('user123', {'logged_in' => true})
decision = user.decide('flag_1')
decision.enabled

// Decide if user sees a feature flag variation
let user = optimizely.createUserContext(userId: "user123", attributes: ["logged_in":true])
let decision = user.decide(key: "flag_1")
let enabled = decision.enabled

For more detailed examples of each SDK, see:

Android SDK example usageAndroid SDK example usage

Go SDK example usageGo SDK example usage

C# SDK example usageC# SDK example usage

Flutter SDK example usageFlutter SDK example usage

Java SDK example usageJava SDK example usage

JavaScript SDK example usageJavaScript SDK example usage – SDK versions 6.0.0 and later.

JavaScript (Browser) SDK example usageJavaScript (Browser) SDK example usage – SDK versions 5.3.5 and earlier.

JavaScript (Node) SDK example usageJavaScript (Node) SDK example usage – SDK versions 5.3.5 and earlier.

PHP SDK example usagePHP SDK example usage

Python SDK example usagePython SDK example usage

React SDK example usageReact SDK example usage

React Native SDK example usageReact Native SDK example usage

Ruby SDK example usageRuby SDK example usage

Swift SDK example usageSwift SDK example usage

Adapt the integration code in your application. Based on the boolean value your application receives, show or hide the flag's functionality for a given user ID.

The goal of the Decide method is to separate the process of developing and releasing code from the decision to turn a flag on. The value this method returns is determined by your flag rules. For example, the method returns false if the current user is assigned to a control or "off" variation in an experiment.

Remember, a user is evaluated against each flag rule in an ordered ruleset before being bucketed into a given rule variation or not. See Interactions between flag rulesInteractions between flag rules for information.

Start your rule and flag (ruleset)

Note For a comparison between the old and new flag management UI, see New flag and rule lifecycle management FAQsNew flag and rule lifecycle management FAQs.

After creating your A/B test and implementing the experiment using the decide method, you can start your test.

With the REST API

Enable the flag by using the Enable the Ruleset for a Flag in an EnvironmentEnable the Ruleset for a Flag in an Environment endpoint.

Endpoint – POST https://api.optimizely.com/flags/v1/projects/PROJECT_ID/flags/FLAG_KEY/environments/ENVIRONMENT_KEY/ruleset/enabled

Authentication – Include your API key in the Authorization header as a Bearer token. See Generate tokens and use the REST APIsGenerate tokens and use the REST APIs.

Request example

curl --request POST \
     --url https://api.optimizely.com/flags/v1/projects/PROJECT_ID/flags/FLAG_KEY/environments/ENVIRONMENT_KEY/ruleset/enabled \
     --header 'accept: application/json' \
     --header 'authorization: Bearer TOKEN'

Response example

{
  "url": "/projects/PROJECT_ID/flags/scheduled_changes_flag/environments/development/ruleset",
  "update_url": "/projects/PROJECT_ID/flags/scheduled_changes_flag/environments/development/ruleset",
  "disable_url": "/projects/PROJECT_ID/flags/scheduled_changes_flag/environments/development/ruleset/disabled",
  "rules": {
    "a-b-test": {
      "key": "a-b-test",
      "name": "a-b-test",
      "description": "",
      "variations": {
        "off": {
          "key": "off",
          "name": "Off",
          "percentage_included": 5000,
          "variation_id": 1502017,
          "id": 6694877
        },
        "on": {
          "key": "on",
          "name": "On",
          "percentage_included": 5000,
          "variation_id": 1502018,
          "id": 6694878
        }
      },
      "baseline_variation_id": 1502017,
      "type": "a/b",
      "distribution_mode": "manual",
      "id": 1599695,
      "urn": "rules.flags.optimizely.com::1599695",
      "archived": false,
      "enabled": false,
      "created_time": "2025-09-15T19:52:05.908420Z",
      "updated_time": "2025-09-16T13:26:07.391036Z",
      "audience_conditions": [],
      "audience_ids": [],
      "percentage_included": 10000,
      "metrics": [
        {
          "event_id": 30097930171,
          "event_type": "custom",
          "scope": "visitor",
          "aggregator": "unique",
          "winning_direction": "increasing",
          "display_title": "New event with properties"
        },
        {
          "event_id": 27189690020,
          "event_type": "click",
          "scope": "visitor",
          "aggregator": "unique",
          "winning_direction": "increasing",
          "display_title": "Add to cart clicks"
        }
      ],
      "allow_list": {},
      "group_rule": {
        "group_id": 0,
        "traffic_allocation": 0
      },
      "group_remaining_traffic_allocation": 100,
      "layer_id": 9300001987253,
      "layer_experiment_id": 9300002732965,
      "status": "paused"
    }
  },
  "rule_priorities": [
    "a-b-test"
  ],
  "id": 1882548,
  "urn": "rulesets.flags.optimizely.com::1882548",
  "archived": false,
  "enabled": true,
  "updated_time": "2025-09-16T13:26:07.391036Z",
  "flag_key": "scheduled_changes_flag",
  "environment_key": "development",
  "environment_name": "Development",
  "environment_id": 24540890184,
  "default_variation_key": "off",
  "default_variation_name": "Off",
  "revision": 7,
  "status": "running",
  "role": "admin"
}

Launch your A/B test by enabling the rule. Use the Update the Ruleset for a Flag in an EnvironmentUpdate the Ruleset for a Flag in an Environment endpoint and setting the enabled path to true.

Endpoint – PATCH https://api.optimizely.com/flags/v1/projects/PROJECT_ID/flags/FLAG_KEY/environments/ENVIRONMENT_KEY/ruleset

Authentication – Include your API key in the Authorization header as a Bearer token. See Generate tokens and use the REST APIsGenerate tokens and use the REST APIs.

Request example

curl --request PATCH \
     --url https://api.optimizely.com/flags/v1/projects/PROJECT_ID/flags/scheduled_changes_flag/environments/development/ruleset \
     --header 'accept: application/json' \
     --header 'authorization: Bearer TOKEN' \
     --header 'content-type: application/json-patch+json' \
     --data '
[
   {
      "op": "replace",
      "path": "/rules/a-b-test/enabled",
      "value": true
   }
]
'

Response example

{
  "url": "/projects/PROJECT_ID/flags/scheduled_changes_flag/environments/development/ruleset",
  "update_url": "/projects/PROJECT_ID/flags/scheduled_changes_flag/environments/development/ruleset",
  "disable_url": "/projects/PROJECT_ID/flags/scheduled_changes_flag/environments/development/ruleset/disabled",
  "rules": {
    "a-b-test": {
      "key": "a-b-test",
      "name": "a-b-test",
      "description": "",
      "variations": {
        "off": {
          "key": "off",
          "name": "Off",
          "percentage_included": 5000,
          "variation_id": 1502017,
          "id": 6694920
        },
        "on": {
          "key": "on",
          "name": "On",
          "percentage_included": 5000,
          "variation_id": 1502018,
          "id": 6694921
        }
      },
      "baseline_variation_id": 1502017,
      "type": "a/b",
      "distribution_mode": "manual",
      "id": 1599695,
      "urn": "rules.flags.optimizely.com::1599695",
      "archived": false,
      "enabled": true,
      "created_time": "2025-09-15T19:52:05.908420Z",
      "updated_time": "2025-09-16T13:28:42.239797Z",
      "audience_conditions": [],
      "audience_ids": [],
      "percentage_included": 10000,
      "metrics": [
        {
          "event_id": 30097930171,
          "event_type": "custom",
          "scope": "visitor",
          "aggregator": "unique",
          "winning_direction": "increasing",
          "display_title": "New event with properties"
        },
        {
          "event_id": 27189690020,
          "event_type": "click",
          "scope": "visitor",
          "aggregator": "unique",
          "winning_direction": "increasing",
          "display_title": "Add to cart clicks"
        }
      ],
      "fetch_results_ui_url": "https://app.optimizely.com/v2/projects/24566260139/results/9300001987253/experiments/9300002732965?baseline=1502017",
      "allow_list": {},
      "group_rule": {
        "group_id": 0,
        "traffic_allocation": 0
      },
      "group_remaining_traffic_allocation": 100,
      "layer_id": 9300001987253,
      "layer_experiment_id": 9300002732965,
      "status": "running"
    }
  },
  "rule_priorities": [
    "a-b-test"
  ],
  "id": 1882548,
  "urn": "rulesets.flags.optimizely.com::1882548",
  "archived": false,
  "enabled": true,
  "updated_time": "2025-09-16T13:28:42.276687Z",
  "flag_key": "scheduled_changes_flag",
  "environment_key": "development",
  "environment_name": "Development",
  "environment_id": 24540890184,
  "default_variation_key": "off",
  "default_variation_name": "Off",
  "revision": 8,
  "status": "running",
  "role": "admin"
}

You can also see step seven (Launch experiment) in the Run experiment recipeRun experiment recipe. See the REST API cookbookREST API cookbook for information on configuring the recipe and step-by-step instructions.

With the Feature Experimentation UI

See Start your rule and flag (ruleset)Start your rule and flag (ruleset) in the user documentation for information on using the UI.

Test with flag variables

After you run a basic "on/off" A/B test, you can increase the power of your experiments by adding remote feature configurations or flag variables.

Flag variations enable you to avoid hard-coding variables in your application. Instead of updating the variables by deploying, you can edit them remotely in the Optimizely app. For information about flag variations, see flag variationsflag variations.

To configure an A/B test with multiple variations:

Create and configure a basic A/B test. See previous steps.

Create flag variationsCreate flag variations containing multiple variables.

Integrate the example codeIntegrate the example code with your application.