Build
...
System tools
System tools for Feature Experimentation
9 min
Mark (Optimizely Agent Platform) includes system tools that Optimizely provides and maintains. System tools are built-in features that help Mark take action. Each tool performs a specific task, such as creating a campaign, uploading files, or generating images. Think of tools like attachments on a Swiss Army knife. Each one has a distinct purpose that helps you get work done.
In addition to the system tools available in Mark, Optimizely Feature Experimentation includes a set of system tools designed to help you improve your experimentation program.
Note
Ask Mark what tools it has at any time. For example, enter "Please list the tools you have with a brief description of what they do and the parameters" into Mark chat.
Click a tool's name to expand it and learn when to use it, its required and optional parameters, and example prompts for calling the tool. If you do not provide a required parameter, Mark prompts you for it.
Experimentation context
Note
- Mark cannot look up detailed results for a specific experiment you ask about by name. However, it can show your top-performing and underperforming experiments with their lift and significance data. For full experiment results, use the Optimizely Experiment Results page.
- The feature is not connected to the Optimizely Analytics product. Mark does not have access to Analytics explorations or custom analyses.
- Results data support is still evolving. While Mark can surface high-level performance information, deeper results integration is still in progress.
- When to use
- If you are unsure what entities (like flags, rules, or environments) are available in your Feature Experimentation setup.
- Before building a query with the exp_execute_query tool, you need to know the exact field names and their types for the entities you are interested in.
- To see how different pieces of your Feature Experimentation data are connected.
- Parameters
- entities – A list of entity types for which you want to retrieve schemas. For Feature Experimentation, common entities include flag, rule, environment, attribute, and event.
- (Optional) include_dependencies – A boolean value that, when set to true, includes schemas for related entities in the response.
- Example prompts
- What are the available fields for feature flags?
- Show me the schema for rules in Feature Experimentation.
- List all the properties of an environment entity.
- I need to understand the structure of attributes used in my feature flags. Get me the schema for attribute.
- When to use
Important
Before using the exp_execute_query tool, you should use the exp_get_schemas tool to understand the available fields and structure for the entities you want to query. This ensures you build your query template correctly.
- Retrieve a list of feature flags, rules, or other Feature Experimentation entities.
- Filter entities based on specific conditions (for example, flags in a particular environment or rules with certain traffic allocation).
- Get detailed information about specific fields of an entity.
- Query across all projects or within a specific project.
- Parameters
- template – A complete query template that defines the steps, filters, and fields you want to retrieve. The template is written in a specific format that outlines how to fetch and structure the data.
- (Optional) project_id – If you want to query data within a specific project, provide its ID. If you do not provide a project_id, the query attempts to run across all projects you have access to. You can often find the project_id in your Optimizely URL (for example, https://app.optimizely.com/v2/projects/1234567890/...).
- Example prompts
- Show me all feature flags in my current project.
- List all rules for the 'new_checkout_flow' feature flag.
- What are the events associated with my feature tests in the 'development' environment?
- Give me a summary of all attributes used in my Feature Experimentation projects.
- Find all feature flags that are currently enabled and their traffic allocation.
The tool returns detailed information for each experiment, including the following:
- Layer Name – The name of the experiment (corresponding to a rule within a flag).
- Project ID – The identifier of the project the feature flag belongs to.
- Variation Name - The specific variation of the feature that achieved the lift.
- Metric Name – The metric used to measure the impact of the experiment.
- Relative Lift Value – The numerical lift value (can be positive for winning or negative for losing).
- Is Significant – Indicates whether the result is statistically significant.
- Winning Direction – The direction of the winning variation (for example, "increasing").
- Start Date – When the experiment began.
- Last Modified – When the experiment was last updated.
Note
The exp_program_reporting_top_experiments tool does not include results data for experiments that use stats accelerator as the traffic distribution mode.
- When to use
- Discover which feature variations are driving the most significant positive outcomes for your product.
- Learn from successful experiments to inform future product design and development decisions.
- Pinpoint features or variations that are having a negative impact, letting you quickly iterate or rollback.
- Demonstrate the value of specific feature releases or experiments to product managers and stakeholders.
- Answer questions like the following:
- "Which feature flags had the biggest impact on user engagement last quarter?"
- "Show me the top 5 winning variations for our 'new_search_algorithm' test"
- "What were the most impactful experiments in project X this year?"
- Parameters
- date_range – The inclusive start and end timestamps in ISO-8601 format (for example, YYYY-MM-DDTHH:MM:SSZ or with offset). This defines the period over which experiments are evaluated.
- (Optional) direction – Specifies whether to rank by "winning" (highest positive lifts) or "losing" (largest negative lifts).
- (Optional) metric_name – The name of the metric to use for calculating lift. If omitted, it defaults to the primary metric.
- (Optional) page_size – The maximum number of experiments to return. Must be between 1 and 100 (inclusive). If omitted, the backend applies its own default.
- (Optional) project_ids – A list of numeric project IDs to filter the results by. If omitted or empty, the tool may aggregate across all authorized projects.
- Example prompts
- Show me the top 10 winning feature experiments for the 'user_onboarding' flow in the last 3 months.
- Which feature rollouts had the biggest negative impact on our 'purchase_conversion' metric this quarter?
- List the top 5 feature variations that increased 'feature_X_usage' in project ID 987654321 between July and September 2025.
- What were the most impactful tests this year, ranked by 'subscription_rate'?
Note
The exp_program_reporting_underperforming_experiments tool does not include results data for experiments that use stats accelerator as the traffic distribution mode.
- When to use
- Find experiments that are not reaching statistical significance, have confidence intervals that include zero, or show negative lift.
- Quickly pinpoint tests that you should stop to free up traffic and resources for more promising ideas.
- Analyze historical data to understand common pitfalls or patterns in underperforming tests.
- Check the health of your running experiments to intervene if they are clearly not progressing towards a positive outcome.
- Parameters
- date_range – Specifies the inclusive start and end timestamps in ISO-8601 format (for example, {"start": "2025-01-01T00:00:00Z", "end": "2025-12-31T23:59:59Z"}). This defines the period for which experiments are analyzed.
- (Optional) project_ids – A list of specific project IDs to filter the experiments. If omitted, the tool aggregates data across all authorized projects.
- (Optional) page_size – The maximum number of underperforming experiments to return (between 1 and 100).
- (Optional) metric_name – The name of the metric to filter by. If omitted, it defaults to the primary metric on the backend. Examples include "primary", "Add to Cart", or "conversion_rate".
- Example prompts
- Show me all underperforming A/B tests from the last quarter.
- Which experiments should I stop or review from the last 6 months for Project X?
- Find experiments with low significance for the 'Add to Cart' metric this year.
- List all experiments that are not performing well between January 1, 2025, and June 30, 2025.
- Are there any underperforming tests in my 'Website Redesign' project?
Note
The exp_program_reporting_win_rate tool does not include results data for experiments that use stats accelerator as the traffic distribution mode.
The win rate is defined as the following:
- Wins – Feature experiments (rules within flags) that show a positive and statistically significant result for the chosen metric.
- Total – All feature experiments that have concluded or been paused within the defined scope.
- Win rate – \text{Win Rate} = \left( \frac{\text{Wins}}{\text{Total}} \right) \times 100
- When to use
- Understand the overall success rate of your feature testing and release process.
- Monitor how your team's ability to launch winning features evolves over time.
- Provide a clear, high-level metric to product leadership and other stakeholders on the effectiveness of your Feature Experimentation program.
- Answer questions like the following:
- "What is the win rate for our feature experiments?"
- "How many of our tests resulted in a positive outcome last year?"
- "What is the win rate for features impacting our core business metrics?"
- Parameters
- date_range – The inclusive start and end timestamps in ISO-8601 format (for example, YYYY-MM-DDTHH:MM:SSZ or with offset). This defines the period over which the win rate is calculated.
- (Optional) project_ids – A list of numeric project IDs to filter the experiments by. If omitted or empty, the tool may aggregate across all authorized projects.
- (Optional) metric_name – The name of the metric used to determine what constitutes a "win" (for instance, a positive and significant result). If omitted, it defaults to the primary metric configured on the backend.
- Example prompts
- What was the win rate for our experiments in the last fiscal year?
- Calculate the win rate for all feature tests in project ID 123456789 between January 1st, 2025, and June 30th, 2025.
- Show me the win rate for feature experiments focused on the 'subscription_upgrade' metric this quarter.
- What is the overall win rate for our Feature Experimentation program across all projects for the past 12 months?
Flag ideation
Note
- You must provide a clear hypothesis for your experiment. A hypothesis is a testable statement that explains what you expect to happen. See Design an effective hypothesis.
- This tool can often derive your project_id and flag_key from your current context (for example, the URL you are viewing). If this information is not available, Mark prompts you to provide it.
- When to use
- Define the specific variables that are part of an experiment.
- Translate a hypothesis into concrete, testable variables for your feature.
- Configure the different elements of your feature for A/B testing.
- Parameters
- hypothesis – The testable statement that explains what you expect to happen in your experiment.
- project_id – The identifier of the Feature Experimentation project where the feature flag is defined.
- flag_key –The key for the feature flag you are modifying.
- Example prompts
- I want to suggest flag variables for an experiment. My hypothesis is, "If we change the call-to-action text on the homepage button, more users will click through to the signup page."
- Generate variables for a feature flag in project 123 and flag new_checkout_flow. The hypothesis is that a simplified checkout form will reduce cart abandonment.
- Suggest variables for testing different pricing tiers. My hypothesis is that offering a 'premium' tier will increase average order value.
Note
- You must provide a clear hypothesis for your experiment. A hypothesis is a testable statement that explains what you expect to happen. See Design an effective hypothesis.
- This tool can often derive your project_id and flag_key from your current context (for example, the URL you are viewing). If this information is not available, Mark prompts you to provide it.
- When to use
- Define the different variations that are part of an experiment.
- Translate a hypothesis into concrete, testable variations for your feature.
- Set up the different configurable experiences of your feature for A/B testing or gradual rollout.
- Parameters
- hypothesis– The testable statement that explains what you expect to happen in your experiment.
- project_id – The identifier of the Feature Experimentation project where the feature flag is defined.
- flag_key – The key for the feature flag you are modifying.
- Example prompts
- I want to suggest flag variations for an experiment. My hypothesis is: "If we change the call-to-action text on the homepage button, more users will click through to the signup page."
- Generate variations for a feature flag in project 123 and flag new_checkout_flow. The hypothesis is that a simplified checkout form will reduce cart abandonment.
- Suggest variations for testing different pricing tiers. My hypothesis is that offering a 'premium' tier will increase average order value.
Feature governance
These tools power the Feature Experimentation Governance agent.
- When to use
- Review flag health, stale flags, and cleanup opportunities across a project without a manual audit.
- Generate the report before you resolve findings with the exp_flag_governance_actions tool.
- Parameters
- project_name – Full or partial name of the Feature Experimentation project to analyze.
- (Optional) project_id – The ID of the project. If more than one project matches the name, Mark asks you to choose one and uses its ID.
- Example prompts
- Generate a governance report for the 'Web Platform' project.
- Which flags in my 'Mobile App' project are stale?
- When to use
- Display the governance report generated by the exp_generate_governance_report tool on a canvas.
- Parameters
- name – The title of the canvas. The agent uses the title from the governance report.
- mime_type – The type of content. The agent uses text/html.
- content – The content to display. The agent passes the governance report HTML without changes.
- Example prompts
- Generate a governance report for the 'Web Platform' project.
- When to use
- Conclude and deploy A/B tests.
- Conclude rollouts that reach 100% of traffic.
- Promote rules to production.
- Archive flags.
- Mark flags as permanent.
- Parameters
- project_id – The ID of the project.
- project_name – The name of the project.
- report_fingerprint – The fingerprint from the governance report. The card uses it to detect changes made after Mark generated the report.
- (Optional) actions – The type of action to show, for example, conclude. If omitted, the card shows all action types.
- Example prompts
- Show me the A/B tests that are ready to conclude and deploy.
- Show me only the archive actions.
- When to use
- Review which flags you marked as permanent.
- Remove the permanent mark from a flag.
- Parameters
- project_id – The ID of the project.
- project_name – The name of the project.
- Example prompts
- Show me the permanent flags in my 'Web Platform' project.
Note
If you use Opti ID, administrators can turn off generative AI in the Opti ID Admin Center. See Turn generative AI off across Optimizely applications.