Turn agent feedback and rephrased questions into an evaluation dataset¶

Mine thumbs-down feedback and user rephrases from agent traces, confirm each with one LLM call, and write a deduplicated eval dataset.

This recipe is a reference implementation: review the code and adapt it to your environment before production use.

Real users tell you where an agent falls short in two ways: they click thumbs down, or they ask the same thing again in different words. This recipe reads both signals from the agent’s observability events, confirms each candidate with a single LLM call, drafts a self-contained test question with ground-truth criteria, drops near-duplicates, and writes EVAL_DATASET in the INPUT_QUERY / GROUND_TRUTH shape that agent evaluations runs against.

When to use this¶

• You have an agent in use and want evaluation questions that reflect where it actually fails, not only questions you thought of in advance.
• You want the mining to be a rerunnable SQL job with every stage inspectable.

When not to use this:

• You need to score answers: that is EXECUTE_AI_EVALUATION; see agent evaluations.
• You need latency or per-request debugging: see diagnose-agent-requests.
• You cannot grant a role access to conversation text. Without READ UNREDACTED AI OBSERVABILITY EVENTS TABLE, questions and feedback text read as NULL and there is nothing to mine.

How it works¶

1. Two signals. Explicit: a CORTEX_AGENT_FEEDBACK event with positive = false, linked to the turn it rates. Implicit: a follow-up turn in a thread whose question re-asks the previous one. A turn’s parent_message_id equals the previous turn’s message_id, so pairs join exactly.
2. Similarity is a pre-filter, not the decision. AI_SIMILARITY runs once per pair and is stored. A real rephrase (“How is Tofu doing?” then “I meant total Tofu revenue by region…”) scores about 0.59 and a genuine follow-up (“What was total revenue in the North region?” then “And how does that compare to the South?”) about 0.42, so no single threshold separates them. The floor only limits how many pairs reach the LLM.
3. One LLM call per candidate. A single AI_COMPLETE with a JSON schema returns whether the pair is dissatisfaction, a self-contained eval_question, an intent, ground_truth_output criteria, and expected_tools. Rows the model answers badly stay in EVAL_CANDIDATES as invalid_llm_output instead of disappearing. That depends on AI_SQL_ERROR_HANDLING_USE_FAIL_ON_ERROR = FALSE: with TRUE, one failed call aborts the whole CREATE TABLE AS, so 05_mine_signals.sql sets it to FALSE for its own session.
4. Dedupe by meaning. AI_EMBED embeds each kept question once; a candidate whose cosine similarity to a higher-priority one is at or above 0.92 is dropped.
5. Drafts, not truth. Ground truth is written as criteria (“must filter to Tofu, cover July 2025 to June 2026, one figure per region”), and every row carries review_status: draft for a human to check before the dataset is used.

Prerequisites¶

Quickstart¶

1. sql/01_setup.sql as ACCOUNTADMIN: role (granted to you), warehouse, database.
2. sql/02_grant_unredacted.sql as ACCOUNTADMIN: lets the recipe role read conversation text.
3. sql/03_agent.sql: 800 synthetic Northwind orders, a semantic view, and ORDERS_ASSISTANT.
4. sql/04_simulate_traffic.sql: replays 8 scripted conversations (13 turns) through DATA_AGENT_RUN, one thread per conversation, logging every call in RUN_LOG.
5. Optional: notebooks/01_submit_feedback.ipynb sends the thumbs up or down in FEEDBACK_SCRIPT through the Feedback REST API.
6. sql/05_mine_signals.sql: builds AGENT_TURNS, FEEDBACK_EVENTS, TURN_PAIRS, EVAL_CANDIDATES, CANDIDATE_EMBEDDINGS, and EVAL_DATASET, then prints a funnel.
7. Review EVAL_CANDIDATES, then run the dataset with agent evaluations.
8. sql/99_teardown.sql when done.

Walkthrough¶

Traffic with threads, in SQL¶

body := TO_JSON(OBJECT_CONSTRUCT(
    'thread_id', thread_id,                 -- NULL on the first turn, so OBJECT_CONSTRUCT omits it
    'parent_message_id', parent_id,
    'messages', ARRAY_CONSTRUCT(OBJECT_CONSTRUCT('role', 'user',
        'content', ARRAY_CONSTRUCT(OBJECT_CONSTRUCT('type', 'text', 'text', question))))));
SELECT TRY_PARSE_JSON(SNOWFLAKE.CORTEX.DATA_AGENT_RUN(:AGENT_FQN, :body, TRUE)) INTO :resp;
thread_id := resp:metadata:thread_id::NUMBER;
parent_id := resp:metadata:assistant_message_id::NUMBER;

The third argument creates a thread when the body has none. The response metadata carries the thread id and assistant message id for the next turn. Each call is in its own exception block that writes the error to RUN_LOG and skips the rest of that conversation.

Explicit feedback¶

The Feedback REST API body has orig_request_id, positive, feedback_message, categories, and thread_id. The notebook finds each turn’s request id in the trace (the trace’s message_id equals the assistant message id returned by DATA_AGENT_RUN), posts the feedback, and records every HTTP status in FEEDBACK_LOG.

On the read side, the documented filter is RECORD:name = 'CORTEX_AGENT_FEEDBACK'; the documentation does not specify where inside the event the fields live. 05_mine_signals.sql therefore matches them by name across both VALUE and RECORD_ATTRIBUTES, links by request id, and falls back to the latest earlier turn in the same thread. The funnel reports feedback it could not link.

Rephrase pairs¶

SELECT cur.request_id, prev.question AS prev_question, cur.question,
       AI_SIMILARITY(prev.question, cur.question) AS similarity
FROM AGENT_TURNS cur
JOIN AGENT_TURNS prev ON prev.thread_id = cur.thread_id AND prev.message_id = cur.parent_message_id
WHERE cur.parent_message_id > 0;

One call per candidate¶

AI_COMPLETE($model,
    '<instructions> ... SIGNAL:\n' || TO_JSON(OBJECT_CONSTRUCT('source', s.source,
        'previous_question', s.prev_question, 'previous_answer', LEFT(s.prev_answer, 2000),
        'question', s.question, 'answer', LEFT(s.answer, 2000), 'feedback_message', s.feedback_message)),
    model_parameters => {'temperature': 0, 'max_tokens': 800},
    response_format  => {'type': 'json', 'schema': {...is_dissatisfaction, reason, eval_question,
                                                     intent, ground_truth_output, expected_tools...}})

The conversation goes in as a JSON document and the instructions tell the model to treat it as data. The output is stored, never executed.

Using the dataset¶

EVAL_DATASET has the two columns an evaluation config maps:

dataset:
  dataset_type: "CORTEX AGENT"
  table_name: "COOKBOOK_AGENT_FEEDBACK_TO_EVALS.APP.EVAL_DATASET"
  dataset_name: "ORDERS_MINED_DATASET"
  column_mapping:
    query_text: "INPUT_QUERY"
    ground_truth: "GROUND_TRUTH"

GROUND_TRUTH contains ground_truth_output (read by answer_correctness) and expected_tools, plus intent, source, source_request_id, and review_status for your own filtering. Staging the config and running EXECUTE_AI_EVALUATION is covered in agent evaluations.

Verify¶

The funnel at the end of 05_mine_signals.sql, for the scripted traffic without the notebook, looks like:

If you ran the notebook, FEEDBACK_LOG has three rows with their HTTP status, feedback events counts what arrived, and each linked thumbs-down becomes an explicit_negative candidate. Check feedback not linked to a turn: it should be 0.

Gotchas¶

Teardown¶

Run sql/99_teardown.sql as ACCOUNTADMIN. It drops the role (which removes the unredacted grant), the database (agent, semantic view, data, and every mined table), and the warehouse. Events already written stay in SNOWFLAKE.LOCAL.AI_OBSERVABILITY_EVENTS, which can’t be modified.

Adapt this recipe¶

Keep these, so a customized version keeps the recipe’s security model:

• Least privilege: ACCOUNTADMIN only for the scripts listed under Privileges; everything else runs as the recipe’s own role.
• No grants to PUBLIC.
• Caller’s rights: procedures run as EXECUTE AS CALLER, so each user’s own grants apply.
• Credentials: create keys, tokens, and secrets yourself, in a separate terminal, and read them from environment variables. Never put them in a script or give them to an agent.
• Dynamic SQL, if you add any, uses bind variables or an allowlist of identifiers, never text concatenated from user or model input.

Change these to fit your use case:

• Object names. Rename them in every script, including teardown, then check that teardown still removes everything.
• Roles, personas, and users: map them to your own, each granted no more than the recipe grants.
• Data: point the scripts at your tables instead of the sample data, then run Verify again.
• Models and warehouse sizes.

To have a coding agent adapt it:

Adapt the agent-feedback-to-evals recipe in agent-feedback-to-evals/ (unzipped from agent-feedback-to-evals.zip) to <describe your use case>.
Read README.md first. Keep everything the "Adapt this recipe" section says to keep.
Change only names, roles and personas, data, and models.
Show me each changed file before you run anything, then run the recipe's Verify steps.

Run it with a coding agent¶

Install or enable the bundled agent-studio and cortex-ai-function-studio skills if they are not
already available; use agent-studio for evaluation follow-ups and cortex-ai-function-studio for
AI function cost or model questions. Then, from agent-feedback-to-evals/:
1. Run SELECT CURRENT_ACCOUNT(), CURRENT_REGION(), CURRENT_ROLE(); and show me the result.
   Stop if it is not the account I expect.
2. Run sql/01_setup.sql with snow sql -f (it switches to ACCOUNTADMIN).
3. sql/02_grant_unredacted.sql grants READ UNREDACTED AI OBSERVABILITY EVENTS TABLE ON ACCOUNT,
   an account-level privilege that lets the recipe role read conversation text. Show me the
   statement and wait for my yes before running it.
4. Run sql/03_agent.sql and sql/04_simulate_traffic.sql. Report any RUN_LOG rows whose
   RUN_STATUS is not 'completed'.
5. Ask me whether I want explicit feedback. If I do, I create a programmatic access token for
   my own user and export SNOWFLAKE_PAT myself; never create, print, read, or store the token.
   Then run notebooks/01_submit_feedback.ipynb and report the HTTP status of each row in
   FEEDBACK_LOG. Otherwise skip the notebook.
6. Run sql/05_mine_signals.sql and show me the funnel and the rows of EVAL_CANDIDATES with
   llm_status, is_dissatisfaction, reason, and eval_question.
7. When I say so, run sql/99_teardown.sql and confirm SHOW DATABASES, SHOW WAREHOUSES and
   SHOW ROLES LIKE 'COOKBOOK_AGENT_FEEDBACK_TO_EVALS%' return nothing.
Never drop or alter objects that do not start with COOKBOOK_.

Related skills¶

• agent-studio (bundled with Cortex Code): building and running agent evaluations and the agent-observability-report for aggregate monitoring. Use it with the dataset this recipe writes.
• cortex-ai-function-studio (bundled): choosing, testing, and estimating the cost of the AI functions used here (AI_COMPLETE, AI_SIMILARITY, AI_EMBED).

References¶

• Monitor Cortex Agent requests
• Feedback REST API
• Use threads with the Cortex Agent REST API
• DATA_AGENT_RUN
• AI_COMPLETE
• AI_SIMILARITY
• AI_EMBED
• Cortex Agent evaluations