Cortex Agent evaluations¶

Access control requirements¶

Access to conduct a Cortex Agent evaluation requires the following:

• Access to a role with the permissions needed for agent observability. This role is used for running agent evaluations, and must have the following:

  • The permissions required for agent monitoring:

    • DATABASE ROLE SNOWFLAKE.CORTEX_USER

    • APPLICATION ROLE SNOWFLAKE.AI_OBSERVABILITY_EVENTS_LOOKUP

    • EXECUTE TASK ON ACCOUNT

    For more information on the agent monitoring permissions, see Snowflake AI Observability reference – Access control.

  • The CREATE FILE FORMAT ON SCHEMA permission on the schema containing your agent.

  • The CREATE DATASET ON SCHEMA permission on the schema containing your agent.

  • The ability to impersonate the user who will run an evaluation. This must be the same as the user logged in through Snowsight who creates and runs the evaluation. For example, to enable impersonation of a user my_user for the role my_observability_role:

    GRANT IMPERSONATE ON USER my_user TO ROLE my_observability_role;
    

    Copy

• The CREATE TASK permission on the schema containing your agent.

• The OWNERSHIP or MONITOR privilege on your agent.

Prepare an evaluation dataset¶

Before starting a Cortex Agent evaluation, you need to prepare a dataset for your evaluation to run against. This dataset consists of an input query column of type VARCHAR that represents your query, and an output column of type OBJECT that contains a description of expected agent behavior. This single column is your ground truth for evaluation.

This object has two keys:

• ground_truth_output – The expected final output of the agent, used in answer correctness evaluation. This value is evaluated on a semantic match, where agent answers that align more closely with your ground truth output receive higher scores.

• ground_truth_invocations – The value of this key is an ARRAY containing JSON objects describing a tool invocation, used for the tool selection and tool accuracy metrics. Use the empty array [] to verify that no tools are called by the agent.

{
  "tool_name": "my_analyst",
  "tool_output": {
    "SQL": "SELECT * FROM EXPECTED_TABLE"
  }
}

{
  "tool_name": "my_search",
  "tool_output": {
    "search results": [
      "context searched 1",
      "context searched 2",
      "context searched N"
    ]
  }
}

For example, if you expect the agent to call get_weather with the inputs of city = "San Francisco" and date == 08/02/2019 and return with a VARIANT of {"temp": "14", "units": "C"}, you would define an entry in the expected outputs array as:

{
  "tool_name": "get_weather",
  "tool_input": {"city": "San Francisco", "date": "08/02/2019"},
  "tool_output": {"temp": "14", "units": "C"}
}

The following example includes the above expected tool invocation as part of a full ground truth entry, where the expected answer from your agent is The temperature was 14 degrees Celsius in San Francisco on August 2nd, 2019.

{
  "ground_truth_output": "The temperature was 14 degrees Celsius in San Francisco on August 2nd, 2019.",
  "ground_truth_invocations": [
      {
        "tool_name": "get_weather",
        "tool_input": {"city": "San Francisco", "date": "08/02/2019"},
        "tool_output": {"temp": "14", "units": "C"}
      }
  ]
}

To bring your JSON evaluation dataset into a Snowflake table, use the PARSE_JSON SQL function. The following example creates a new table agent_evaluation_data for the evaluation dataset, and inserts a row for the input query What was the temperature in San Francisco on August 2nd 2019? with the ground truth JSON from the previous example:

CREATE OR REPLACE TABLE agent_evaluation_data (
    input_query VARCHAR,
    ground_truth OBJECT
);

INSERT INTO agent_evaluation_data
  SELECT
    'What was the temperature in San Francisco on August 2nd 2019?',
    PARSE_JSON('
      {
        "ground_truth_output": "The temperature was 14 degrees Celsius in San Francisco on August 2nd, 2019.",
        "ground_truth_invocations": [
            {
              "tool_name": "get_weather",
              "tool_input": {"city": "San Francisco", "date": "08/02/2019"},
              "tool_output": {"temp": "14", "units": "C"}
            }
        ]
      }
    ');

Start an agent evaluation¶

Begin your evaluation of a Cortex Agent by doing the following:

1. Sign in to Snowsight.

2. In the navigation menu, select AI & ML » Agents.

3. Select the agent you want to conduct an evaluation of.

4. Select the Evaluations tab.

5. Select New evaluation run.

   The New evaluation run modal opens.

6. In the Name field, provide a name for your evaluation. This name should be unique for the agent being evaluated.

7. Optional: In the Description field, provide any comments for the evaluation.

8. Select Next.

   This advances to the Select dataset modal.

9. Select the dataset used to evaluate your agent. You can choose either Existing dataset or Create new dataset.

   To use an existing dataset:

   1. From the Database and schema list, select the database and schema containing your dataset.

   2. From the Select dataset list, select your dataset.

   To create a new dataset:

   1. From the Source table - Database and schema list, select the database and schema containing the table you want to import to a dataset.

   2. From the Select source table list, select your source table.

   3. From the New dataset location - Database and schema list, select the database and schema to place your new dataset.

   4. In the Dataset name field, enter your dataset name. This name needs to be unique among the schema-level objects in your selected schema.

10. Select Next.

    This advances to the Select metrics modal.

11. From the Input query list, select the column of your dataset which contains the input queries.

12. For each of the Metrics, change the toggle to active for any metric you want included in your evaluation. Select the column of your dataset containing the ground truth for your evaluation.

13. Select Create to create the evaluation and begin the evaluation process.

At any point, you can select Cancel to cancel creating the evaluation, or select Prev to return to the previous modal.

Inspect evaluation results¶

Evaluation results include information about the requested metrics, details of the agent’s threads of reasoning, and information about the LLM planning stage for each executed trace in the thread. The Evaluations tab for an agent in Snowsight gives you an overview of every evaluation run and its summary results.

To view evaluation results in Snowsight:

1. Sign in to Snowsight.

2. In the navigation menu, select AI & ML » Agents.

3. Select the agent you want to conduct an evaluation of.

4. Select the Evaluations tab.

Identifying and reporting errors¶

While in private preview, Agent evaluation errors may surface as SQL statement errors without information about the underlying issue. In order to get a detailed report of the error and help us improve our evaluation error messages, contact Snowflake Support with the incident number from your query history.

Cost considerations¶

Using Cortex Agent evaluations can incur highly variable cost depending on the evaluation metrics selected, optional evaluation information you supply, the structure of your agent, and other considerations. During private preview please work directly with your Snowflake representative regarding cost.