Streaming Transformations¶

Overview¶

Streaming Transformations let you filter, enrich, reshape, and aggregate data in flight while you ingest it with Snowpipe Streaming. You define the logic once, in the CREATE PIPE statement, and every record is transformed before it lands in your table. This shortens the time from raw data to data that is ready for querying, with typically <10s end-to-end latency and even less for simple transformations targeting Interactive tables. It helps from two directions:

1. If you run a separate stream processing system in front of Snowflake, you may no longer need it. That lowers total cost of ownership and improves freshness.
2. If you transform data after it lands in Snowflake, transforming at ingestion is, in relevant cases, faster and simpler, because there is no separate post-landing transformation layer to build, schedule, and maintain.

Transformations land into standard Snowflake tables, Interactive tables, and managed Iceberg tables and are available in all Snowflake deployments.

Transformations¶

Streaming Transformations are embedded directly in the COPY INTO part of a CREATE PIPE definition:

1. Stream filtering — a WHERE clause that selectively ingests records matching a predicate.
2. Stream enrichment — one or more LEFT OUTER JOIN clauses that enrich each ingested record with columns from a dimension table.
3. Custom scalar logic — SQL and JavaScript scalar UDFs called from the SELECT list or the WHERE predicate, for example to map a site code to its local timezone, normalize a unit, format strings, or classify a raw value into a named tier.
4. Lateral Flatten — expands an array in the record into one row per element, so a single streamed record lands as multiple rows.
5. Insert-triggered materialized views — keep filtered projections or aggregates of ingested rows up to date in separate materialized views that are maintained as data is ingested.

Projection transformations, including XML and JSON processing, continue to work through the pipe subquery described in CREATE PIPE and COPY INTO <table>.

What you can do with Streaming Transformations¶

Each use case below has a scenario that shows the transformations to combine, with SQL, sample input, and result.

1. AI agent observability — turn agent telemetry into one row per tool call. Filter out internal traffic, attach model provider and pricing, and estimate the cost of every call.
2. Observability and OpenTelemetry — land logs and traces in query-ready tables. Drop noisy debug records, expand attributes into rows, attach the owning team and service, and keep error and count views up to date.
3. Industrial IoT telemetry — turn batched device messages into one typed, named reading per row, mapped from raw tag IDs to engineering names and units, with per-minute rollups next to the raw data.
4. Security and incident event management (SIEM) — keep only the event types that matter, enrich them with asset and threat-intelligence context, classify severity with a UDF, and fan out one row per owning team for alert routing.
5. Clickstream and product analytics — tie each event to account tier and region, parse user agents and URLs with a UDF, and keep fresh event counts per tier and region.
6. Privacy and compliance — hash or mask personal data before it lands, and ingest only records with consent, so raw identifiers never reach the table.

Pages in this section¶

1. Pipe Creation
2. Stream filtering
3. Stream enrichment
4. Custom scalar logic
5. Lateral Flatten
6. Insert-triggered materialized views
7. Reference
8. Scenarios
   1. Scenario A: AI agent observability
   2. Scenario B: Observability logs and OpenTelemetry
   3. Scenario C: Industrial IoT telemetry
   4. Scenario D: Security events
   5. Scenario E: Product and clickstream events
   6. Scenario F: Privacy-safe ingestion
9. Monitoring & Troubleshooting

See also¶

1. CREATE PIPE
2. COPY INTO <table>
3. Snowpipe Streaming with high-performance architecture
4. Transform data during a load
5. Querying metadata for staged files
6. NOT NULL, PRIMARY KEY, UNIQUE, and FOREIGN KEY constraints

Documentation update history¶