---
title: Run A/B tests in Feature Experimentation
slug: feature-experimentation/docs/run-a-b-tests-in-feature-experimentation
docTags: 
createdAt: 2026-08-05T20:39:57.000Z
---

- Optimizely Feature Experimentation

If you are new to experimentation, you can get a lot done with a simple `ON` or `OFF` A/B test. This configuration has one flag with two variations:

- One "flag\_on" variation.
- One "flag\_off" variation.

## Restrictions

If you plan to have multiple experiments and flag deliveries (targeted deliveries) in your flag, the experiment must be the first rule in your ruleset for the flag.

## Configuration overview

To configure a basic A/B test, complete the following steps:

1. (Prerequisite) [Create a flag](docId\:eH-UMm_rOn_GdGDGLV6__).
2. (Prerequisite) [Handle user IDs](docId\:j0PPoMJWYbz04K76EKLUI).
3. [Create](docId:8pnfCv-sR37goLcxTtCPv) an **A/B Test** rule in the Optimizely app. 
4. [Configure](docId:8pnfCv-sR37goLcxTtCPv) the **A/B Test** rule.
5. [Integrate the example decide code](docId:8pnfCv-sR37goLcxTtCPv) that the Optimizely Feature Experimentation app generates with your application.
6. [Start](docId:8pnfCv-sR37goLcxTtCPv) your A/B rule and flag (ruleset)
7. [Test](docId:8pnfCv-sR37goLcxTtCPv) your experiment in a non-production environment. See [QA and troubleshoot](docId\:JU1Sh9PVLBMIW2_VDbTjx) for information.
8. Discard any test user events and enable your experiment in a production environment.

## Create an experiment

### Create an A/B Test rule

1. Select your flag from the **Flags** list.
2. Select the [environment](docId\:YpSOMYlWkUccLCL-uB-ZE) you want to target.
3. Click **Add Rule**.
4. Select **A/B Test**.

![](https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/Lyi1ZfdKV8C1TJsUoVtfb_ruleset-add-rule-menu-with-ab-test-selected.png)

### Configure your A/B test rule

1. Enter a **Name**
2. The **Key** is automatically created based on the **Name**. You can optionally update it.
3. (Optional) Click **Add description** to add a description of the rule.
4. (Optional) Click **Add Hypothesis** to add the experiment reasoning, the expected outcome, and why you expect it. Click **Save Hypothesis**.
5. (Optional) Click **Add Test Plan** to add how a rule runs, including duration, success criteria, and rollout strategy. You can attach the plan as a URL or upload it as a file. Click **Save Test Plan.**
6. (Optional) Search for and add audiences. To create an audience, see [Target audiences](docId\:UDBvr5a8WHNkyOqzPhOr9). Audiences evaluate in the order in which you drag and drop them. You can choose whether to match each user on any or all of the audience conditions.
7. Set the **Traffic Allocation** to assign a percentage of your audience to bucket into the experiment.

:::hint{type="info"}
If you plan to change the traffic's **Traffic Allocation** after running the experiment or select **Stats Accelerator** for the **Distribution Mode**, you need to implement a user profile service before starting the experiment.

For information, see [Ensure consistent user bucketing using a user profile service](docId\:KxcJHu_E0VyZ2X3jV3WH5).
:::

8. Add **Metrics** based on tracked user events. See [Manage events in Feature Experimentation](docId\:BuRedAbq5odrXzImb_plY) for information on how to create events using the UI or [Create events](docId\:v2goO-UxyIA4AtTSR-h-9) for how to use the Feature Experimentation REST API.
   - For information about selecting metrics, see [Choose metrics in Feature Experimentation](docId\:e2qGf29S6pO8SW90_IVuH). 
   - For instructions on creating metrics, see [Create a metric in Optimizely using the metric builder](docId\:cphstZe3xKhsY7WsnnLZK).
9. Choose how your audience is distributed using **Distribution Mode**. Use the drop-down list to select one of the following:
   - **Manual** – Variations are given equal traffic distribution. If you select this option, you can adjust the distribution percentages for each variation in the following step.
   - **Stats Accelerator** – Manipulates the traffic distribution automatically to minimize time to [statistical significance](https://www.optimizely.com/optimization-glossary/statistical-significance/). For information, see [Stats Accelerator](docId\:NmxCi10mrJY5_LouQUpNF).
10. Choose the flag variations to compare in the experiment. See [Create flag variations in the UI](docId\:nmipY0LAkbpzTcsggjlPn) or [Create flag variations using the REST API](docId\:nmipY0LAkbpzTcsggjlPn) for information.. 
    - For a basic experiment, you can include one variation in which your flag is on and one in which your flag is off.
    - For a more advanced A/B/n experiment, create variations with multiple flag variables. See the [Test with flag variables](docId:8pnfCv-sR37goLcxTtCPv) section.
11. (Optional) Click **More (...)** for a variation and select **Set as baseline**. The baseline is the "original" or "control" variation against which Feature Experimentation compares the other variations to measure performance. If you do not select a baseline, Feature Experimentation arbitrarily selects a variation as the baseline.

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/CkaZONBSaazFHJo7LAl21_variations-set-as-baseline-option-in-variation-menu.png" size="40" isUploading="false" width="805" height="264" darkWidth="805" darkHeight="264" showCaption="false"}

:::Paragraph{indent="1"}
After you set a baseline variation, to update it, complete the following:
:::

:::Paragraph{listStyleType="disc" indent="2"}
Click **More (...)** for the baseline variation and select **Unset baseline** to clear it.
:::

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/h5-4x8exkztgPA016EFI8_variations-unset-baseline-option-in-variation-menu.png" size="40" isUploading="false" width="822" height="372" darkWidth="822" darkHeight="372" showCaption="false"}

:::Paragraph{listStyleType="disc" indent="2"}
Click **More (...)** for a variation and select **Set as baseline** to replace the baseline variation with this variation.
:::

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/SK01cRaVEjrgFTBN4P8oc_variations-set-as-baseline-on-second-variation.png" size="40" isUploading="false" width="818" height="452" darkWidth="818" darkHeight="452" showCaption="false"}

12. (Optional) Click **Allowlist: Force up to 50 users into any variation(s)** and enter the User ID. See [Allowlisting](docId:5UlaFy9bllyJz-PW39UtC).
13. (Optional) Add the experiment to an **Exclusion Group**.
14. Enter any **Custom Field** information, if available. See [Custom fields in Feature Experimentation](docId\:FR469ek790vzzg2Pwpuev).
15. Click **Save**.

![](https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/Qj9z1qjy99SyCIR_ENa1o_ruleset-ab-test-rule-configuration-panel.png)

## Run a pre-launch experiment review with Mark

Before implementing your A/B test rule, you can use [Optimizely Agent Platform](docId\:IzAAN1fw7whD1HB4iODAB) to review your experiment configuration and recommend changes to maximize your odds of reaching statistical significance. Before doing so, [Configure Feature Experimentation for Optimizely Agent Platform](docId\:ttsUwHVA7PytfZEdvOtiJ).

1. Select your rule.
2. Click **Review Experiment**.

![](https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/2C1pjIQCFo2FTaH-3_3sf_ruleset-review-experiment-button-in-rule-panel.png)

Review and apply Mark's feedback before you run your A/B test rule to ensure the best results.

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/ISneSdfNbQQ1ifeu5RFKJ_recommended-action-items-opal-experiment-review-results.png" size="50" isUploading="false" width="1090" height="1410" darkWidth="1090" darkHeight="1410" showCaption="false"}

See [Test Optimizely Experimentation](docId\:JU1Sh9PVLBMIW2_VDbTjx) for additional information on QA steps in Optimizely Experimentation.

/genera

:::hint{type="info"}
**Note**

If you use Opti ID, administrators can turn off generative AI in the Opti ID Admin Center. See [Turn generative AI off across Optimizely applications](docId\:X5U0kKD2azV5cIdNqwGNj).
:::

## Implement the experiment using the decide method

### Flag is implemented in your code

If you have already implemented the flag using a `Decide` method, you do not need to take further action (Optimizely Feature Experimentation SDKs are designed so you can reuse the exact flag implementation for different flag rules).

### Flag is not implemented in your code

If the flag is not implemented yet, see [Run A/B tests](docId:8pnfCv-sR37goLcxTtCPv) in the developer documentation for code samples.

For more detailed examples of each SDK, see, the following:

- [Android SDK example usage](docId\:nQkw1CJn6rhkkZcHUhWlu)
- [Go SDK example usage](docId\:BtqVa7hPVV3lakICHOePE)
- [C# SDK example usage](docId\:TcOrbpenEwK6XfJSXEWhQ)
- [Flutter SDK example usage](docId\:aZh7xqKhZ9b8I15ukdQnS)
- [Java SDK example usage](docId\:oczD_Ka_bhgCygo60xG31)
- [JavaScript SDK example usage](docId\:Hd7ZwchAg70iG5S_5kgpM) – SDK versions 6.0.0 and later.
- [Javascript (Browser) SDK example usage](docId\:c8V_lO-12gUfsBNUuczJI) – SDK versions 5.3.5 and earlier.
- [JavaScript (Node) SDK example usage](docId:0lhn1UwOiZajgk6h-cq2a) – SDK versions 5.3.5 and earlier.
- [PHP SDK example usage](docId\:QDWBJTJc3c2qnd5EhAPQ9)
- [Python SDK example usage](docId:0Afx9oaNuXQo-439j2JzL)
- [React SDK example usage](https://docs.developers.optimizely.com/feature-experimentation/docs/example-usage-react)
- [React Native SDK example usage](https://docs.developers.optimizely.com/feature-experimentation/docs/example-usage-reactnative)
- [Ruby SDK example usage](docId\:Oh9NVDFfn7X-s1WTLfPUY)
- [Swift SDK example usage](docId:3ZVLXuKme4kiqdO2Yt_H1)

Adapt the integration code in your application. Based on the Boolean value your application receives, show or hide the flag's functionality for a given user ID.

The goal of the `Decide` method is to separate the process of developing and releasing code from the decision to turn a flag on. The value this method returns is determined by your flag rules. For example, the method returns false if the current user is assigned to a control or "off" variation in an experiment.

Remember, a user evaluates each flag rule in an ordered ruleset before being bucketed into a given rule variation or not. See [Interactions between flag rules](docId:_4OpAAw99L5HVhT-OCJTW) for information.

### Understand the Off variation

In Feature Experimentation, the **Off** variation is a unique state where you explicitly disable the feature for users. Unlike other variations (such as a, b, c, d) that typically return `enabled: true`, the **Off** variation always returns `enabled: false`.

However, this does not exclude users from tracking. Users assigned to the **Off** variation are still tracked, which you can verify through the `ruleKey` in the decision response.

### Variation behaviors

**Off variation**

- `enabled` **response** – `false`
- **Included in datafile** – No
- **Access method** – REST API only

**Baseline (non-Off) variation**

- `enabled` **response** – `true`
- **Included in datafile** – Yes
- **Access method** – Datafile and REST API

The `decision.enabled` flag does not indicate whether a variation is a baseline. Baselines are not included in the datafile but are accessible through the REST API.

If you do not use the **Off** variation, `feature.enabled` always returns `true`. Conversely, if you assign the **Off** variation, it returns `false`.

The **Off** variation affects tracking and decision responses. Although **Off** is excluded from the datafile, it remains part of the experiment logic and is accessible through API calls.

## Start your rule and flag (ruleset)

For information between the old and new flag management UI in Optimizely Feature Experimentation, see [New flag and rule lifecycle management FAQs](docId\:Bog7o25N6pDj3-2qpsVGn).

After [creating your A/B test](#) and [implementing the experiment using the decide method](#), you can start your test.

1. Set your rule as **Ready to run**. Click **Run** on your A/B test rule.

![](https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/e4goRbrG28_cDIy4tOCXp_ruleset-run-button-on-ab-test-rule.png)

2. Click **Ok** on the **Ready to Run Status** page. This alerts you that your rule is set to **Ready to run** until you start your ruleset (flag).

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/BB6LNan1XuqUA5A5WML7g_ready-to-run-ready-to-run-confirmation-dialog.png" size="40" isUploading="false" width="744" height="443" darkWidth="744" darkHeight="443" showCaption="false"}

3. Click **Run** to start your flag and rule.

![](https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/eKmhxpdFZ9iDtlkikpeKn_ruleset-ruleset-run-button-with-rule-ready-to-run.png)

Congrats! Your A/B test is running.

## Test with flag variables

When you have run a basic "on/off" A/B test, you can increase the power of your experiments by adding remote feature configurations or flag variables.

Flag variations let you avoid hard-coding variables in your application. Instead of updating the variables by deploying, you can edit them remotely in the Optimizely Feature Experimentation UI. See [Flag variations and flag variables overview](docId\:m5k7AZmkz0gSS20cE3rRi).

To configure an A/B test with multiple variations, complete the following:

1. Create and configure a basic A/B test. See previous steps.
2. Create flag variations containing multiple variables.
3. [Integrate the example code](docId\:nmipY0LAkbpzTcsggjlPn) with your application.