Cortex Analyst evaluations¶

How verified queries are used during evaluation¶

When you select verified queries for an evaluation run, Cortex Analyst creates a temporary copy of your semantic view with those selected queries removed. Cortex Analyst then generates SQL using this temporary copy, which does not contain the evaluation queries. This prevents the evaluation queries from influencing SQL generation, ensuring that the evaluation measures how well Cortex Analyst can answer questions without relying on exact matches from verified queries.

Verified queries that you do not select for evaluation remain in the temporary semantic view and continue to guide Cortex Analyst during the evaluation run, just as they would during normal usage.

Snowsight¶

Begin your evaluation of a semantic view by doing the following:

1. Sign in to Snowsight.

2. In the navigation menu, select AI & ML » Cortex Analyst.

3. From the list, select the semantic view you want to run the evaluation on.

4. Select the Evaluations tab.

5. Select Create evaluation run.

6. In the Name field, provide a name for your evaluation. This name should be unique for the semantic view being evaluated.

7. Select Next.

   This advances to the Select verified queries modal.

8. Select which verified queries to include in the evaluation. You can either select all verified queries or select a specific set by checking the corresponding boxes.

9. Select Run evaluation.

SQL¶

Cortex Analyst evaluation runs can also be started with SQL using the EXECUTE_AI_EVALUATION function. This function accepts the following evaluation_job values:

• 'START': Start an evaluation run.

• 'STATUS': Query the progress of an evaluation run.

• 'CANCEL': Cancel a running evaluation.

• 'DELETE': Delete a completed evaluation run and its results.

Each call requires the following additional arguments:

• run_parameters: A SQL OBJECT containing the key run_name, with a value of the name of your run.

• config_file_path: A stage file path pointing to your run configuration YAML file. For the YAML specification, see Analyst evaluation YAML specification.

The following example starts an evaluation run called Evaluation run 1:

CALL EXECUTE_AI_EVALUATION(
  'START',
  OBJECT_CONSTRUCT('run_name', 'Evaluation run 1'),
  '@EVAL_DB.EVAL_SCHEMA.METRICS/analyst_evaluation_config.yaml'
);

After a run starts, you can query its progress:

CALL EXECUTE_AI_EVALUATION(
  'STATUS',
  OBJECT_CONSTRUCT('run_name', 'Evaluation run 1'),
  '@EVAL_DB.EVAL_SCHEMA.METRICS/analyst_evaluation_config.yaml'
);

To cancel or delete a run, replace 'STATUS' with 'CANCEL' or 'DELETE'.

Snowsight¶

The Evaluations tab for a semantic view in Snowsight gives an overview of every evaluation run and a summary of each, including the number of query regressions.

To view evaluation results:

1. Sign in to Snowsight.

2. In the navigation menu, select AI & ML » Cortex Analyst.

3. From the list, select the semantic view you want to view evaluations for.

4. Select the Evaluations tab.

5. Select an individual run to see detailed results.

The run detail page shows:

• Accuracy – The percentage of verified queries where the generated SQL was judged correct, with an option to Improve the semantic view.

• Regressions – The number of verified queries that were previously correct but are now failing.

• Latency – Average and per-query response times for Cortex Analyst.

• Per-query results – For each verified query: the natural language question, the expected SQL, the generated SQL, and whether the result was correct or incorrect. Select a query to see the detailed comparison.

SQL¶

To retrieve the results of an evaluation run, use the GET_ANALYST_AI_EVALUATION_DATA function. This function has the following required arguments:

• database: The database containing the semantic view.

• schema: The schema containing the semantic view.

• object_name: The name of the semantic view.

• object_type: The string constant 'SEMANTIC VIEW'.

• run_name: The name of the evaluation run to retrieve.

The following example displays the full evaluation details for a run called Evaluation run 1, where the semantic view is named SEMANTIC_VIEW_EVAL stored on the schema EVAL_DB.EVAL_SCHEMA:

SELECT * FROM TABLE(SNOWFLAKE.LOCAL.GET_ANALYST_AI_EVALUATION_DATA(
  'EVAL_DB',
  'EVAL_SCHEMA',
  'SEMANTIC_VIEW_EVAL',
  'SEMANTIC VIEW',
  'Evaluation run 1')
);

YAML format¶

evaluation:
  analyst_params:
    analyst_name: "SEMANTIC_VIEW_EVAL"
    analyst_type: "SEMANTIC VIEW"
  source_metadata:
    type: "verified_queries"
    # Optional: list specific verified queries by their question text.
    # If omitted, all verified queries are used.
    verified_queries:
      - "What are our top 10 customers from the last 30 days?"
      - "What is the total revenue by region for Q1 2025?"

metrics:
  - "sql_correctness"

analyst_params

• analyst_name: The name of the semantic view to run the evaluation against.

• analyst_type: The string constant SEMANTIC VIEW.

source_metadata

• type: The type of source used as the evaluation data. For Cortex Analyst, the only supported source type is verified_queries.

• verified_queries (optional): A list of questions matching the question field of each verified query that should be used as ground truth for the evaluation. If not provided, all verified queries are used.

metrics

The metrics to compute for the evaluation. sql_correctness is the only supported metric for Cortex Analyst evaluations.

Upload configuration to a stage¶

Upload your YAML configuration to a Snowflake stage. The following example creates a file format, creates a stage, and uploads a local configuration file:

CREATE OR REPLACE FILE FORMAT evals_db.evals_schema.yaml_file_format
  TYPE = 'CSV'
  FIELD_DELIMITER = NONE
  RECORD_DELIMITER = '\n'
  SKIP_HEADER = 0
  FIELD_OPTIONALLY_ENCLOSED_BY = NONE
  ESCAPE_UNENCLOSED_FIELD = NONE;

CREATE OR REPLACE STAGE evals_db.evals_schema.metrics
  FILE_FORMAT = evals_db.evals_schema.yaml_file_format;

PUT file:///Users/dev/analyst_evaluation_config.yaml @evals_db.evals_schema.metrics
  AUTO_COMPRESS='false'
  OVERWRITE=TRUE;