Two- and three-stage testing in Feature Experimentation
- Optimizely Feature Experimentation
Optimizely Feature Experimentation comes with two environments automatically: Production and Development. When you select a flag, the Development environment ruleset displays.
You can add additional environments and change which environment is set as primary. See Manage environmentsManage environments information.
Remember that each environment has a different datafile, with which you will initialize the Optimizely Feature Experimentation SDKs.
Note
You do not necessarily need separate environments to make use of Optimizely Feature Experimentation's Environments feature. You can, for example, create an environment for users with a test cookie. You will just need to implement the logic that would initialize Optimizely using the test cookie environment when the user has your test cookie.
Two-stage and three-stage test environments
Here are two examples of environment use cases. You can have an unlimited number of environments, and there are many more possibilities for using them than we have described here. Use as many environments as your workflow requires.
In general, the fewer environments you use, the simpler the workflow.
Two-stage "sandbox" QA
The two-stage "sandbox" QA use case is a common setup in web development and involves the following separate, sequential environments:
- Local – Code is running only for the development team and QA.
- Production – Code is deployed and running for all visitors.
You can use the "sandbox" approach even if you do not have separate deployment environments for your application. One common use case is to toggle between the local and production files for debugging. For example, you could load the local file when a certain “test” cookie is enabled, when a certain URL parameter is attached (like ?debug=true), or based on some other check that matches your internal workflow.
Three-stage testing
The three-stage testing use case involves these separate, sequential environments:
- Local – Code is running only for the developer who is writing it. You do not need to create a separate Local environment for each developer, just one environment where the experiment will run during development.
- Staging – Code is shared with the larger development team and QA.
- Production – Code is deployed and running for all visitors.
Example test environment scenario
Imagine you have three experiments: sort-by-name, show-subtotal, and warn-empty-cart. You set up a new environment named Testing so you can test your experiment code before you deploy it in the Production environment. Suppose that:
- You completed testing for the sort-by-name experiment and deployed it.
- You are currently testing the show-subtotal experiment.
- Your warn-empty-cart experiment was tested but has not yet been deployed.
The following table is an overview of the statuses of your three experiments in your two environments:
Experiment | Testing environment activity state and datafile status | Production environment activity state and datafile status |
|---|---|---|
sort-by-name | Paused | Running |
show-subtotal | Running | Not started |
warn-empty-cart | Paused | Not started |
You can keep experiments running in your testing environment that are also running in production if you prefer.
Now, imagine that the project owner creates a third environment: Staging. Optimizely Feature Experimentation will generate an additional datafile named Staging. Every experiment is in the “Not started” activity state in Staging, and the Staging datafile will show the experiment status as “Not started.”
When your experiment code is fully functional and ready to test, switch it to the Testing environment by pointing your code to the Testing datafile.