Observability for Cortex AI Gateway¶

Cortex AI Gateway records traces and spans for all requests traversing the gateway. Traces give you full execution transparency into agent behavior: which models were invoked, step timing, token consumption, and failure reasons.

Query traces with the AGENT_TRACE_TABLE table function:

-- Retrieve request and response records for the SNOWFLAKE gateway
SELECT * FROM TABLE(AGENT_TRACE_TABLE('SNOWFLAKE'));

Querying AGENT_TRACE_TABLE requires the MONITOR privilege on the gateway object. See Access control.

How traces are structured¶

The gateway models AI activity using standard OpenTelemetry execution hierarchies:

• Conversation thread (conversation_id): an optional identifier grouping related multi-turn interactions over time.
• Trace (trace_id): a single end-to-end execution, or agent turn, which can encompass multiple model calls.
• Span (span_id): a single atomic unit of work within a trace, such as one inference call. Each span records timing, status, the model requested, token counts, and outcome metadata.

Each span is one row. Group rows by trace_id to assemble a trace, and a conversation_id groups related traces into one thread:

conversation_id          Optional. Groups related turns in one thread.
    └── trace_id         One agent turn, which may encompass multiple model invocations.
        └── span         operation = 'chat'          An inference call.

Every inference span has gen_ai.operation.name set to chat.

To inspect traces, spans, and conversations without assembling any of this yourself, use the AI Gateway Observability page in Snowsight.

Send client traces to the gateway¶

The spans the gateway records are what it sees from the outside: one span per request it serves. A client that uses OpenTelemetry can also export its own traces to the gateway, which adds the client’s view of the same work, including the steps it took between requests.

The gateway accepts OTLP traces over http/protobuf or http/json at <gateway-endpoint>/telemetry/v1/traces, authenticated with the same programmatic access token you use for inference. Get the endpoint from SHOW AI GATEWAYS, as described in Gateway endpoint.

Client-exported spans land in AGENT_TRACE_TABLE alongside the spans the gateway records, so a single query covers both.

The attributes a client exports are the client’s own, so they vary by client and aren’t described here. When the client propagates its trace context on the requests it sends, its spans share a trace_id with the gateway’s, so grouping on it reassembles a turn from both sides. See Trace context propagation.

What the exporter has to be set to¶

Whatever client you’re configuring, its OTLP trace exporter needs these three values:

A client that reads the standard OpenTelemetry environment variables takes these as OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, and OTEL_EXPORTER_OTLP_TRACES_HEADERS, with OTEL_TRACES_EXPORTER set to otlp. Clients that keep telemetry behind a switch of their own need that turned on as well.

Example: OpenCode¶

OpenCode exports through an OpenTelemetry plugin, configured in ~/.config/opencode/opencode.json:

{
  "plugin": [
    [
      "@devtheops/opencode-plugin-otel",
      {
        "enabled": true,
        "protocol": "http/protobuf",
        "endpoint": "<gateway-endpoint>/telemetry",
        "otlpHeaders": "Authorization=Bearer <SNOWFLAKE_PAT>",
        "tracePropagationProviders": ["snowflake-cortex"]
      }
    ]
  ]
}

endpoint takes the base, and the exporter appends the OTLP path, so requests arrive at <gateway-endpoint>/telemetry/v1/traces.

Trace context propagation¶

For client spans and gateway spans to join into one turn, the client has to send a traceparent header on its requests. Without it both sides are still recorded, they just don’t share a trace_id.

For how to turn propagation on when setting up a coding agent, see Instrument a coding agent.

What the gateway records¶

Traces are stored in a Snowflake event table, so they use the standard event table schema: identifiers and timing are top-level columns, while semantic detail lives in the scope, record, record_attributes, and resource_attributes objects, which you extract with the colon operator and cast. For the full schema, see Event table columns. Because every gateway row is a span, record_type is always SPAN. For the fields a span carries, see the SPAN record type.

Span identity and outcome¶

Gateway and user¶

These live in resource_attributes, because they describe the process that emitted the span rather than the individual request.

Request and response¶

HTTP¶

Query traces¶

Two queries cover most of what you need: one that lists turns, and one that opens a turn up into its spans.

Neither filters on scope, so both cover client-exported spans as well as the gateway’s own. Add scope:"name"::string = 'aigateway/tracing' to either to see only what the gateway recorded.

List turns¶

Each span is one row, so this rolls the rows of a trace into a single row per turn:

WITH ev AS (
  SELECT trace:"trace_id"::string                                       AS trace_id,
         trace:"span_id"::string                                        AS span_id,
         record_attributes:"gen_ai.request.model"::string               AS model,
         record_attributes:"gen_ai.input.messages"::string              AS llm_input,
         record_attributes:"gen_ai.conversation.id"::string             AS conversation_id,
         resource_attributes:"user"::string                             AS user_name,
         TRY_TO_NUMBER(record_attributes:"http.status_code"::string)     AS http_status,
         start_timestamp,
         timestamp
  FROM TABLE(AGENT_TRACE_TABLE('SNOWFLAKE'))
  WHERE timestamp >= '<start>' AND timestamp < '<end>'
)
SELECT trace_id,
       MIN(start_timestamp)                                          AS started_at,
       DATEDIFF('millisecond', MIN(start_timestamp), MAX(timestamp)) AS duration_ms,
       COUNT(*)                                                      AS span_count,
       MAX(user_name)                                                AS user_name,
       MIN_BY(model, IFF(model IS NULL, NULL, timestamp))            AS model,
       COUNT(DISTINCT model)                                         AS model_count,
       MAX(conversation_id)                                          AS conversation_id,
       MIN_BY(llm_input, IFF(llm_input IS NULL, NULL, timestamp))    AS first_input,
       IFF(COUNT_IF(http_status >= 400) = 0, 'SUCCESS', 'FAILURE')   AS status
FROM ev
GROUP BY trace_id
ORDER BY started_at DESC
LIMIT 10000;

Inspect one turn¶

Take a trace_id from the previous query. One row per span means no rollup is needed:

SELECT trace:"span_id"::string                                              AS span_id,
       record:"name"::string                                                AS span_name,
       record_attributes:"gen_ai.operation.name"::string                    AS operation,
       start_timestamp                                                      AS span_start,
       DATEDIFF('millisecond', start_timestamp, timestamp)                  AS duration_ms,
       DATEDIFF('millisecond', MIN(start_timestamp) OVER (), start_timestamp) AS offset_ms,
       record:"status":"code"::string                                       AS status,
       TRY_TO_NUMBER(record_attributes:"http.status_code"::string)          AS http_status,
       record_attributes:"gen_ai.request.model"::string                     AS model,
       TRY_TO_NUMBER(record_attributes:"gen_ai.usage.input_tokens"::string)  AS input_tokens,
       TRY_TO_NUMBER(record_attributes:"gen_ai.usage.output_tokens"::string) AS output_tokens
FROM TABLE(AGENT_TRACE_TABLE('SNOWFLAKE'))
WHERE timestamp >= '<start>' AND timestamp < '<end>'
  AND trace:"trace_id"::string = '<trace_id>'
ORDER BY span_start
LIMIT 2000;

To read the prompts and responses, add the content attributes. They’re large, so select them only when you need them:

record_attributes:"gen_ai.input.messages"        AS input_messages,
record_attributes:"gen_ai.output.messages"       AS output_messages,
record_attributes:"gen_ai.system_instructions"   AS system_instructions

Cost considerations¶

Traces are stored in an event table, so gateway observability is billed the way other telemetry data is: Snowflake charges for the serverless compute used to ingest the traces, and for the storage they occupy in the event table. Refer to the Snowflake Service Consumption Table for the applicable rates. For more detail on how telemetry ingestion is billed and how to keep it down, see Costs of telemetry data collection.

Turning on payload capture records the full prompts and responses, so it increases the volume stored per request.

Usage notes¶

• Individual records are capped at 1 MB. Records larger than that are truncated. Truncation behavior preserves the beginning and the end of the record, and truncates the middle sections.