Cortex Agent evaluations¶

Cortex Code¶

To have Cortex Code assist you with creating a dataset for your evaluation, use the dataset-curation sub-skill of the Cortex Code cortex-agent skill. For more information about Cortex Code skills, see Cortex Code CLI - Skills.

Dataset format¶

The table used to create a dataset for evaluation has an input query column of type VARCHAR that represents your query, and an output column of type VARIANT that contains a description of expected agent behavior. This single output column is used as the ground truth by the LLM judge.

Values in the output column have one key, ground_truth_output. The value of this key is used in answer correctness evaluation. LLM judges use ground truth to evaluate your agent’s output by including it in their prompt.

To bring a JSON dataset into a Snowflake table, use the PARSE_JSON SQL function. The following example creates a new table agent_evaluation_data to use for an evaluation dataset, and inserts a row for the input query What was the temperature in San Francisco on August 2nd 2019? with the ground truth of The temperature was 14 degrees Celsius in San Francisco on August 2nd, 2019..

CREATE OR REPLACE TABLE agent_evaluation_data (
    input_query VARCHAR
);

INSERT INTO agent_evaluation_data
  SELECT
    'What was the temperature in San Francisco on August 2nd 2019?',
    PARSE_JSON('
      {
        "ground_truth_output": "The temperature was 14 degrees Celsius in San Francisco on August 2nd, 2019.",
      }
    ');

Data you provide in the ground_truth column that isn’t used by a selected metric is ignored. When conducting an evaluation run with only reference-free metrics, you can leave the output column empty.

When running your first evaluation, you’ll have the option to create a new dataset from an existing table.

Cortex Code¶

To have Cortex Code run an evaluation, use the evaluate-cortex-agent sub-skill of the Cortex Code cortex-agent skill. For more information about Cortex Code skills, see Cortex Code CLI - Skills.

Snowsight¶

Begin your evaluation of a Cortex Agent by doing the following:

1. Sign in to Snowsight.

2. In the navigation menu, select AI & ML » Agents.

3. Select the agent you want to conduct an evaluation of.

4. Select the Evaluations tab.

5. Select New evaluation run.

   The New evaluation run modal opens.

6. In the Name field, provide a name for your evaluation. This name should be unique for the agent being evaluated.

7. Optional: In the Description field, provide any comments for the evaluation.

8. Select Next.

   This advances to the Select dataset modal.

9. Select the dataset used to evaluate your agent. You can choose either Existing dataset or Create new dataset.

   To use an existing dataset:

   1. From the Database and schema list, select the database and schema containing your dataset.

   2. From the Select dataset list, select your dataset.

   To create a new dataset:

   1. From the Source table - Database and schema list, select the database and schema containing the table you want to import to a dataset.

   2. From the Select source table list, select your source table.

   3. From the New dataset location - Database and schema list, select the database and schema to place your new dataset.

   4. In the Dataset name field, enter your dataset name. This name needs to be unique among the schema-level objects in your selected schema.

10. Select Next.

    This advances to the Select metrics modal.

11. From the Input query list, select the column of your dataset which contains the input queries.

12. For each of the System metrics, change the toggle to active for any metric you want included in your evaluation. Select the column of your dataset containing the ground truth for your evaluation.

13. (Optional) To conduct a custom evaluation, toggle on Custom metrics.

    1. Select the database and schema containing the stage where your custom evaluation configuration is stored.

    2. Select the stage where your custom evaluation configuration is stored.

    3. Select the YAML configuration file for your custom evaluation.

       Note

       In Snowsight, only the custom evaluation definitions are loaded from your YAML configuration. The rest of the YAML file must still be valid. For the evaluation YAML specification, see Agent Evaluation YAML specification.

    4. For each custom metric, change the toggle to active if you want it included in your evaluation. Select the column of your dataset containing the ground truth for this evaluation.

14. Select Create to create the evaluation and begin the evaluation process.

At any point, you can select Cancel to cancel creating the evaluation, or select Prev to return to the previous modal.

SQL¶

To start or retrieve information on an evaluation with SQL, use the EXECUTE_AI_EVALUATION function. This function has the following required arguments:

• evaluation_job: A string value of ‘START’ or ‘STATUS’.

• run_parameters: A SQL OBJECT containing the key run_name, with a value of the name of your run.

• config_file_path: A stage file path pointing to your run configuration YAML file. This path can’t be a signed URL. For the evaluation YAML specification, see Agent Evaluation YAML specification.

Use the evaluation_job value ‘START’ to start an evaluation. The following example starts a run called run-1 using the agent evaluation configuration from @eval_db.eval_schema.metrics/agent_evaluation_config.yaml:

CALL EXECUTE_AI_EVALUATION(
  'START',
  OBJECT_CONSTRUCT('run_name', 'run-1'),
  '@eval_db.eval_schema.metrics/agent_evaluation_config.yaml'
);

After a run starts, you can query its progress with the evaluation_job value ‘STATUS’. This call returns a table in the format used for AI Observability Runs. The following example queries the status of the agent evaluation started from the previous example:

CALL EXECUTE_AI_EVALUATION(
  'STATUS',
  OBJECT_CONSTRUCT('run_name', 'run-1'),
  '@eval_db.eval_schema.metrics/agent_evaluation_config.yaml'
);

Cortex Code¶

Cortex Code offers two sub-skills of the cortex-agent skill. Use the investigate-cortex-agent-evals sub-skill to inspect evaluations and find any issues in your configuration or data. Use the optimize-cortex-agent sub-skill to take results from completed evaluations and improve the performance of your agent.

Snowsight¶

The Evaluations tab for an agent in Snowsight gives you an overview of every evaluation run and its summary results.

To view evaluation results in Snowsight:

1. Sign in to Snowsight.

2. In the navigation menu, select AI & ML » Agents.

3. Select the agent you want to conduct an evaluation of.

4. Select the Evaluations tab.

SQL¶

To retrieve raw evaluation details, use the GET_AI_EVALUATION_DATA (SNOWFLAKE.LOCAL) function. This function has the following required arguments:

• database: The database containing the agent.

• schema: The schema containing the agent.

• agent_name: The name of the agent.

• agent_type: The string constant ‘CORTEX AGENT’. This value is case-insensitive.

• run_name: The name of the evaluation run to retrieve.

This function returns a table of event data described in Evaluation results table format. The following example displays the full evaluation details for a run called run-1, where the agent is named evaluated_agent stored on the schema eval_db.eval_schema:

SELECT * FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_EVALUATION_DATA(
  'eval_db',
  'eval_schema',
  'evaluated_agent',
  'CORTEX AGENT',
  'run-1')
);

Query traces for a single record¶

To access a single record from an evaluation trace, use the GET_AI_RECORD_TRACE (SNOWFLAKE.LOCAL) function. This function has the following required arguments:

• database: The database containing the agent.

• schema: The schema containing the agent.

• agent_name: The name of the agent.

• agent_type: The string constant ‘CORTEX AGENT’. This value is case-insensitive.

• record_id: The record ID to filter by.

This function returns a table of event data described in Evaluation results table format. The following example displays the trace for the record 9346efc3-5dd6-4038-9b1a-72ca3d3b768c, where the agent is named evaluated_agent stored on the schema eval_db.eval_schema:

SELECT * FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_RECORD_TRACE(
  'eval_db',
  'eval_schema',
  'evaluated_agent',
  'CORTEX AGENT',
  '9346efc3-5dd6-4038-9b1a-72ca3d3b768c'
));

Query evaluation errors and warnings for a run¶

To access logs for warnings and errors that happened during an evaluation run, use the GET_AI_OBSERVABILITY_LOGS (SNOWFLAKE.LOCAL) function. This function has the following required arguments:

• database: The database containing the agent.

• schema: The schema containing the agent.

• agent_name: The name of the agent.

• agent_type: The string constant ‘CORTEX AGENT’. This value is case-insensitive.

This function returns a table of event data described in Evaluation results table format. The following example checks for errors and warnings for a run called run-1, where the agent is named evaluated_agent stored on the schema eval_db.eval_schema:

SELECT * FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_LOGS(
  'eval_db',
  'eval_schema',
  'evaluated_agent',
  'CORTEX AGENT')
)
  WHERE TRUE
  AND (record:"severity_text"='ERROR' or record:"severity_text"='WARN')
  AND record_attributes:"snow.ai.observability.run.name"='run-1';

Dataset definition¶

The dataset value defines a new dataset from existing table data, mapping columns for the input query and ground truth. For the structure required for your ground_truth column, see Dataset format. The keys for the dataset value are:

• dataset_type: The string constant “CORTEX AGENT”. This value is case-insensitive.

• table_name: The fully qualified name of the table to use for the dataset’s contents.

• dataset_name: The name of the created dataset.

• column_mapping: The mapping of the required evaluation input column query_text and output column ground_truth to columns of the table to create the dataset from.

The resulting dataset is stored in the same database and schema as the table it’s constructed from.

The following example dataset definition shows a dataset named evaluation_input created from the evals_db.evals_schema.evaluation_data table, using the user_question as input and expected_outcome to define ground truth:

dataset:
 dataset_type: "CORTEX AGENT"
 table_name: "evals_db.evals_schema.evaluation_data"
 dataset_name: "evaluation_input"
 column_mapping:
   query_text: "user_question"
   ground_truth: "expected_outcome"

Agent configuration¶

The evaluation value sets the configuration for the agent to conduct an evaluation against. The keys for the evaluation value are:

• agent_params: A dictionary describing the agent to conduct the evaluation for. This value uses the keys:

  • agent_name: The name of the agent to evaluate.

  • agent_type: The string constant “CORTEX AGENT”. This value is case-insensitive.

• (Optional) run_params: Metadata for identifying this evaluation run. This value uses the keys:

  • (Optional) label: The label for this evaluation.

  • (Optional) description: A detailed description of the evaluation.

• source_metadata: A dictionary describing the dataset used for the evaluation. This value uses the keys:

  • type: The string constant “DATASET”. This value is case-insensitive.

  • dataset_name: The name of the dataset to use.

The following example agent configuration runs an agent named evaluated_agent with the label Basic evaluation, using the dataset evaluation_input:

evaluation:
 agent_params:
   agent_name: "evaluated_agent"
   agent_type: "CORTEX AGENT"
  run_params:
   label: "Basic evaluation"
  source_metadata:
   type: "DATASET"
   dataset_name: "evaluation_input"

Metrics selection¶

The metrics value is a sequence of metrics to evaluate, including your own custom metric definitions. The accepted values for pre-defined metrics are:

• answer_correctness: Measure the agent’s response correctness against a ground truth output.

• logical_consistency: Measure consistency across agent instructions, planning, and tool calls. This metric is reference-free and doesn’t use a dataset.

metrics:
  # Built-in metrics
  - "answer_correctness"
  - "logical_consistency"
  # Custom metric with prompt
  - name: "relevance"
    score_ranges:
      min_score: [1, 3]
      median_score: [4, 6]
      max_score: [7, 10]
    prompt: |
      Evaluate the relevance of the agent's response to the user's query.
      Rate from 1-10 where:
      1 = Completely irrelevant
      4 = Somewhat irrelevant
      6 = Neutral
      8 = Mostly relevant
      10 = Highly relevant and on-topic

      You can compare the {{output}} with the {{ground_truth}} to help you understand if the contents are relevant or not

      Consider:
      - Does the response address the user's question?
      - Is the information provided appropriate to the context?
      - Are there any tangential or off-topic elements?

Full example configuration¶

Combining all of the previous example sections gives a full Agent Evaluation configuration:

# Optional: Create dataset before running evaluation
dataset:
  dataset_type: "CORTEX AGENT"
  table_name: "EVALS_DB.EVALS_SCHEMA.EVALUATION_DATA"
  dataset_name: "EVALUATION_INPUT"
  column_mapping:
    query_text: "user_question"
    ground_truth: "expected_outcome"

# Evaluation task configuration
evaluation:
 agent_params:
   agent_name: "evaluated_agent"
   agent_type: "CORTEX AGENT"
  run_params:
   label: "Basic evaluation"
  source_metadata:
   type: "DATASET"
   dataset_name: "EVALUATION_INPUT"

  # Built-in metrics (simple strings)
  - "answer_correctness"
  - "logical_consistency"

  # Custom metric definition
  - name: "relevance"
    score_ranges:
      min_score: [1, 3]
      median_score: [4, 6]
      max_score: [7, 10]
    prompt: |
      Evaluate the relevance of the agent's response to the user's query.
      Rate from 1-10 where:
      1 = Completely irrelevant
      4 = Somewhat irrelevant
      6 = Neutral
      8 = Mostly relevant
      10 = Highly relevant and on-topic

      You can compare the {{output}} with the {{ground_truth}} to help you understand if the contents are relevant or not

      Consider:
      - Does the response address the user's question?
      - Is the information provided appropriate to the context?
      - Are there any tangential or off-topic elements?

Upload configuration to a stage¶

Agent Evaluation configurations are required to have a specific file format for Snowflake to parse them. The following snippet demonstrates creating the required yaml_file_format on the schema evals_db.evals_schema, then creates the stage evaluation_config to upload an agent configuration to:

CREATE OR REPLACE FILE FORMAT evals_db.evals_schema.yaml_file_format
  TYPE = 'CSV'
  FIELD_DELIMITER = NONE
  RECORD_DELIMITER = '\n'
  SKIP_HEADER = 0
  FIELD_OPTIONALLY_ENCLOSED_BY = NONE
  ESCAPE_UNENCLOSED_FIELD = NONE;

CREATE OR REPLACE STAGE evals_db.evals_schema.evaluation_config
  FILE_FORMAT = evals_db.evals_schema.yaml_file_format;

Upload your configuration to a created stage through Snowsight by navigating to In the navigation menu, select Ingestion » Add Data and selecting Load files into a Stage. You can also use the SQL PUT command to upload a local YAML file. The following example demonstrates copying the local file /Users/dev/evaluation_config.yaml to the stage evals_db.evals_schema.evaluation_config:

PUT file:///Users/dev/evaluation_config.yaml @evals_db.evals_schema.evaluation_config
  AUTO_COMPRESS='false'
  OVERWRITE=TRUE;

If you create your YAML in a Workspace, you can copy it from your active workspace to a stage. The following example copies the file evaluation_config.yaml from your workspace to the stage evals_db.evals_schema.evaluation_config:

COPY FILES INTO @evals_db.evals_schema.evaluation_config
  FROM 'snow://workspace/USER$.PUBLIC.DEFAULT$/versions/live'
  FILES=('custom_metric_config.yaml');