---
title: Configure Output Evaluation
slug: agent-platform/docs/configure-output-evaluation
docTags: 
createdAt: 2026-08-12T15:37:04.000Z
---

Output Evaluation scores each specialized agent run against quality criteria, examples, and a baseline score you define. Use Output Evaluation to measure agent quality consistently as you iterate, without manually reviewing every run.

Configure Output Evaluation per agent on the **Output Evaluation** sub-tab of the **Quality** tab.

Output Evaluation results are independent of Execution Guardrails outcomes. Output Evaluation sometimes marks a run as Passed when Execution Guardrails flags it, and vice versa. See [Configure Execution Guardrails](docId\:c-SaebPySbLwE08gOrngT) for details on guardrail-based scoring.

## Prerequisites

- A specialized agent exists in your Agent Platform instance. See [Create a specialized agent](docId\:tZPJWDyE1GSB-epLsUh5S).
- You have permission to edit the agent.

## Access the Output Evaluation sub-tab

Open the Output Evaluation sub-tab to configure scoring or view results for a specialized agent.

1. Go to [home.optimizely.com](https://home.optimizely.com).
2. Select your organization.
3. Click **Opal**.

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/6jLEWe63JyOlsWmw0H0Ag_optimizely-world-organization-home-product-selector-opal-tile.png" size="100" width="2190" height="1378" isUploading="false" darkWidth="2190" darkHeight="1378" showCaption="false" indent="1"}

4. Click **Agents**.

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/yOfCKLgJlc4xmxsytTvUL_opal-agents-nav-item-in-sidebar.png" size="100" width="2120" height="1250" isUploading="false" darkWidth="2120" darkHeight="1250" showCaption="false" indent="1"}

5. Select the **Your Agents** tab.

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/Mckv-up3ITywFTXbsCzUQ_agents-your-agents-tab-list-view.png" size="100" width="2172" height="1204" isUploading="false" darkWidth="2172" darkHeight="1204" showCaption="false" indent="1"}

6. Select a specialized agent.
7. Click **Quality**.
8. Click **Output Evaluation**.

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/R9lePMqmkQNjQWGjSJw0e_quality-output-evaluation-tab.png" size="100" isUploading="false" width="2134" height="600" darkWidth="2134" darkHeight="600" showCaption="false" indent="1"}

## Configure evaluation criteria

Evaluation criteria are statements that an LLM-as-Judge model applies when scoring each run. An LLM-as-Judge model is a Large Language Model (LLM) trained to evaluate other model outputs against quality standards. Add up to 10 criteria for each agent. Criteria apply to both Output Evaluation Examples and Conversation Evaluation Samples.

1. Click **Add Criterion** in the **Evaluation Criteria** section.
2. Enter the criterion as a clear, testable statement. For example: *Content must be at least 250 words long.* or *Content must not include em dashes.*
3. (Optional) Repeat steps 1–2 for each additional criterion, up to 10.
4. Click **Save** to add your criterion.

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/lEQ25XBahrEbwISC4fyUB_configure-evaluation-criteria.png" size="100" isUploading="false" width="800" height="143" darkWidth="800" darkHeight="143" showCaption="false" indent="1"}

## Add output examples or conversation samples

Output Evaluation displays two sections for every specialized agent: **Output Evaluation Examples (Single-Turn)** and **Conversation Evaluation Samples (Multi-Turn)**. Populate the section that matches your agent's interaction mode (single-shot or multi-turn). Agent Platform ignores entries in the section that does not match the agent's mode.

![](https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/NbKtF4Q0SGuAxPYxe1Sbz_add-output-examples-or-conversation-samples.png)

### Add an output example for a single-turn agent

Output examples are preferred outputs that show what a successful single-turn run looks like. Add up to five.

1. Click **Add Example** in the **Output Evaluation Examples** section.
2. Enter a **Name**.
3. Provide the input variables for the example.
4. Enter the sample output.
5. Click **Add** to save your changes.
6. (Optional) Repeat steps 1–5 for each additional example, up to five total.

### Add a conversation sample for a multi-turn agent

Conversation samples are reference conversations that show ideal multi-turn behavior. Add up to five.

1. Click **Add Sample** in the **Conversation Evaluation Samples** section.
2. Enter a **Name**.
3. (Optional) Enter a **Description**.
4. Enter the **Turn 1** information, including the **User message** and optionally the **Expected assistant response,** **Tool calls**, and **Notes**.
5. (Optional) Click **Add Turn** and repeat the previous step with the turn's information.
6. Click **Add** to save your changes.
7. Repeat steps 1–6 for each additional sample, up to five total.

## Set the baseline evaluation score

The baseline evaluation score is the lowest score that counts as **Passed**.

1. Select 75%, 80%, 85%, 90%, or 95% from the baseline evaluation score drop-down list.

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/IH6TRUzVCulPthRoQeuYb_set-the-baseline-evaluation-score.png" size="100" isUploading="false" width="800" height="65" darkWidth="800" darkHeight="65" showCaption="false" indent="1"}

2. Click **Update** to save your changes.

A lower baseline marks more runs as Passed but accepts more variation in quality. A higher baseline enforces stricter quality but marks more runs as Failed. The default value is 85%.

## How Output Evaluation scores runs

Output Evaluation uses an LLM-as-Judge model to score each agent execution. The model evaluates the run against:

- The evaluation criteria you defined.
- The output examples or conversation samples you provided.
- A default rubric.

Each run produces an Evaluation Score between 0% and 100%. Agent Platform compares the score to the baseline evaluation score:

- **Passed** – The score met or exceeded the baseline.
- **Failed** – The score fell below the baseline.
- **Not Evaluated** – Agent Platform has not yet scored the run.

### Default rubric

Agent Platform scores every agent against a default rubric, even without custom criteria. The default rubric covers:

- **Accuracy** – Whether the output is factually correct based on the agent's inputs and context.
- **Completeness** – Whether the output addresses every part of the task.
- **Format consistency** – Whether the output follows the expected structure or format.
- **Usefulness** – Whether the output is practical and actionable for the task.

When your custom criteria conflict with the default rubric, the custom criteria take priority.

### Multi-turn evaluation

For multi-turn agents, Agent Platform scores the conversation at three distinct levels so you can identify where quality breaks down.

- **Per criterion** – Identifies which quality dimensions are weakest.
- **Per turn** – Evaluates how each individual response performed.
- **Across the conversation** – Measures coherence, goal completion, and efficiency.

## View Output Evaluation results

The Summary card on the Output Evaluation sub-tab displays the following:

- **Total Executions for this version** – The total number of runs for the current agent version.
- **Avg. Output Evaluation Score** – The average score across evaluated runs.
- **Passed** – The count of runs that met or exceeded the baseline.
- **Failed** – The count of runs that fell below the baseline.
- **Not Evaluated** – The count of runs Agent Platform has not yet scored.

![](https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/ynj70DjiwXuxJggk-3RQ0_view-output-evaluation-results.png)

To see individual run scores, go to the **Logs** tab. Each run row in the Logs table shows its **Evaluation Score** and **Status**. Click a run row to open the execution details panel. The panel displays the **Evaluation Score** alongside the **Guardrail Status**.

## Related articles

- [Quality tab overview](docId\:RyEJ8-_Fwgaw2WxYQKt1g)
- [Configure Execution Guardrails](docId\:c-SaebPySbLwE08gOrngT)
- [Execution Advisor](docId\:ruQalAXRuJ5xx-iM1LDeP)

:::hint{type="info"}
**Note**

If you use Opti ID, administrators can turn off generative AI in the Opti ID Admin Center. See [Turn generative AI off across Optimizely applications](docId\:X5U0kKD2azV5cIdNqwGNj).
:::