Guardrail alerts
Guardrail alerts let you configure metric alerts that protect business-important metrics during an experiment. Each alert watches a metric you choose and notifies you when a variation moves that metric past a threshold you define, relative to the baseline. Optimizely sends the alert so you can keep an eye on the metric without constantly returning to the results page.
You can use guardrail alerts to:
- Catch regressions early and decide whether to continue, pause, or adjust the experiment.
- Guard metrics such as revenue, checkout completion, signup rate, session time, or retention.
- Reduce the manual monitoring of running experiments.
- Give your team an early warning when a variation harms a critical key performance indicator (KPI).
How guardrail alerts work
You configure a guardrail alert on a specific metric. For that metric, every variation has an improvement value relative to the baseline, and the alert watches those values.
You define the threshold as a percentage change relative to the baseline. Its sign sets the direction: a negative threshold watches for a drop, and a positive threshold watches for an increase. Optimizely compares the threshold against each variation's improvement and triggers the alert when any variation breaches it.
Optimizely measures improvement relative to the baseline. For example, a baseline of 10% that rises to 11% is a 10% relative improvement.
Consider a metric whose variations show improvements of +20% and +3%, with a threshold of -5%. Neither variation breaches the threshold, so no alert fires. If another variation drops to -6%, it breaches the -5% threshold, and Optimizely notifies you. This protects a metric such as revenue per user. Optimizely notifies you if revenue drops in any variation, so you do not have to check the results page yourself.
The following conditions let you configure how sensitive the alert is:
- Minimum visitor count – Set a minimum visitor count so the alert evaluates only after each variation and the baseline reach that count. Early in an experiment, when the number of visitors is low, metrics are volatile and the improvement swings widely. A higher minimum visitor count prevents alerts before the signal is steady.
- Statistical significance (optional) – Add statistical significance as a third condition. When you turn it on, an alert fires only after a variation breaches the threshold, reaches the minimum visitor count, and reaches statistical significance.
Configure guardrail alerts
You can configure guardrail alerts on a per-metric level. Complete the following steps on the results page:
- Under Advanced Settings, toggle Set alerts on for the metric you want to monitor.
- In Alert when threshold is breached, enter the percentage change that triggers the alert and select a positive or negative value.
- In Alert only if users count is at least, enter the minimum visitor count. Optimizely sends an alert only after the baseline and every variation reach this count. For example, set 10,000 to wait until the estimate is less volatile.
- In Notify, choose the users who receive the alert through email. By default, the field lists the users in your instance. You can leave the field blank and subscribe to receive alerts through Slack.
- (Optional) Select Alert only when Statsig is reached. The alert then fires only when a variation breaches the threshold, meets the visitor count, and reaches statistical significance.
- Click Save.
How you receive alerts
Email alerts
Email alerts go only to the users you select in the Notify field when you configure the guardrail alert. The following is an example of an email alert:
Slack alerts
To receive guardrail alerts in Slack, complete the following steps:
- Go to More (…) > Tools > Apps.
- Add the Optimizely app to Slack.
- Click Login to Experimentation and sign in. After you sign in, the following Slack commands are available:
- /subscribe – Subscribe the current channel to a project.
- /unsubscribe – Unsubscribe from a project.
- /unsubscribe-all – Unsubscribe the channel from all project notifications.
- /show-subscribed-projects – List the experimentation projects the channel subscribes to.
- /optimizely-help – Open the help prompt with command guidance.
- Open the Slack channel where you want to receive alerts. To invite the Optimizely app, type @Optimizely, click Send, then click Add them.
- Type /subscribe and click Send. Optimizely then posts every guardrail alert from every experiment in that project to the channel.
- Use the Project Selection drop-down list to choose the project whose alerts post to the channel.
The Slack channel now receives alerts for every metric with guardrail alerts enabled in your selected project. The following is an example of a Slack alert:
Alert frequency
Optimizely checks guardrail alerts on the following schedule:
- Days 1-14 – Every six hours.
- Days 15-30 – Every 24 hours.
- Days 31-60 – Every 36 hours.
- Days 61-90 – Every 48 hours.
- Days 91-365 – Every 96 hours.
- After day 365 – Optimizely stops checking guardrail alerts.
Automatically pause an experiment
Optimizely can also pause an experiment automatically, so a breaching variation stops serving before it causes further harm.
To turn it on, configure the guardrail alert and select Pause experiment when guardrail is breached. This setting applies per guardrail metric and is off by default.
When a guardrail metric with the auto pause enabled breaches its threshold, Optimizely pauses the experiment. The pause happens during the same check that sends the alert. Guardrail checks run on a schedule rather than continuously. The pause happens at the next scheduled check after the breach, not in real time.
The pause honors the same gating conditions as the alert:
- Alert only if users count is at least – Neither the alert nor the pause fires while the measured visitor count is below this value. The gate applies even when a variation already breached the threshold. For example, a variation breaches the threshold at 400 visitors against a 500-visitor minimum. No pause happens. Optimizely pauses the experiment at the first scheduled check after the count reaches 500.
- Alert only when Statsig is reached – The pause waits for statistical significance, as the alert does.
After an experiment pauses this way, its results page shows an Auto-Paused due to Metric Breach banner. The banner names the guardrail metric that breached its threshold and confirms that Optimizely paused the experiment.