Snowpipe Streaming¶

Snowpipe Streaming is Snowflake’s real-time ingestion service built on our latest high-performance architecture. It enables applications to stream rows directly from devices, applications, and services into Snowflake tables or Snowflake-managed Apache Iceberg tables. This direct path can remove staging files, intermediate object storage, message buses, and connector services that the workload doesn’t otherwise need.

Snowpipe Streaming supports two ingestion modes. In both modes, a channel is a logical path that carries rows through a pipe to a target table:

• Elastic Channels are the recommended starting point for most new applications. Producers write directly without creating or coordinating channels; Snowflake manages them and scales ingestion as traffic changes. An acknowledgement confirms that Snowflake has durably buffered the append, so the producer can release its retained copy; table processing and query visibility follow. Elastic Channels provide at-least-once delivery without an ordering guarantee. Producers retain unacknowledged events according to their delivery requirements.
• Named Channels provide ordered, exactly-once ingestion within each channel by using offset tokens. Use Named Channels when reading from a source that requires strict ordering semantics, such as Kafka partitions or Change Data Capture (CDC).

Snowpipe Streaming delivers:

• Up to 20 GB/s throughput per table
• As low as 5 seconds ingest-to-queryable latency
• Direct ingestion without application-managed channels through Elastic Channels
• Ordered, exactly-once ingestion through Named Channels and offset tokens
• Streaming into Snowflake-managed Apache Iceberg tables

The results you observe depend on workload shape and configuration, including row size, table width (number of columns), SDK buffering or REST request batching, concurrency, table type, transformations, and clustering.

Why use Snowpipe Streaming¶

• Simpler direct ingestion: Elastic Channels let producers stream rows directly from devices and services into Snowflake, reducing pipeline hops without requiring you to create channels or coordinate ingestion across producers. Snowflake scales the ingest path as producers and traffic change.
• Exactly-once and ordered ingestion when required: Named Channels use offset tokens to track committed progress and preserve row order within each channel. They map naturally to source partitions and make strict exactly-once recovery straightforward.
• High throughput, low latency: Designed to support ingest speeds of up to 20 GB/s per table, with ingest-to-queryable latency as low as 5 seconds. Results depend on workload shape and configuration.
• In-flight transformations: Cleanse, reshape, and transform data during ingestion by using COPY command syntax within the PIPE object. Reorder columns, cast types, and apply expressions before data is committed to the target table, with no separate ETL step needed.
• Pre-clustering at ingest time: Sort data during ingestion for optimized query performance on tables with clustering keys.
• Apache Iceberg table support: Stream data into Snowflake-managed Iceberg tables, including both Iceberg v2 and Iceberg v3 tables. For more information, see Snowpipe Streaming high-performance architecture with Apache Iceberg™ tables.
• Schema evolution: Automatically adapt table schemas to changing data structures. Snowflake can add new columns detected in the incoming stream without manual DDL changes.
• Row-level error visibility: Enable error logging to capture rows that fail processing after acknowledgement, with details for diagnosis and recovery.
• Simplified pipelines: With Elastic Channels, producers write rows directly into Snowflake tables or Iceberg tables without staging files or intermediate message-bus infrastructure that the workload doesn’t otherwise need.
• Serverless and scalable: Compute resources scale automatically based on ingestion load. No infrastructure to manage.
• Transparent pricing: Throughput-based billing calculated by credits per uncompressed GB of data ingested. For more information, see Snowpipe Streaming high-performance architecture: Understand your costs.

How to connect¶

Snowpipe Streaming supports multiple ingestion paths to fit different workloads:

The Java, Python, and Node.js SDKs use a shared Rust-based client core. Append rows as they arrive: the SDK automatically buffers and batches appends using time and size thresholds, and handles compression and sending data to Snowflake. Direct REST clients instead group rows into newline-delimited JSON (NDJSON), with one JSON object per line, and handle compression themselves.

To get started, choose Elastic Channels or Named Channels, then follow the SDK or REST tutorial in that section.

For a side-by-side comparison and use-case guidance, see Choosing a channel type.

For technical details about the PIPE object, channels, offset tokens, and supported data types, see Key concepts.

Recommended for¶

• High-volume streaming workloads with per-table throughput needs of up to 20 GB/s
• Real-time analytics and dashboards with ingest-to-queryable latency as low as 5 seconds
• IoT, telemetry, and distributed applications using Elastic Channels through an SDK or the REST API
• CDC pipelines using Named Channels with exactly-once delivery guarantees
• Apache Kafka topic ingestion using the Snowflake Connector for Kafka
• Streaming into Apache Iceberg tables for open table format analytics

Snowpipe Streaming versus Snowpipe¶

Snowpipe Streaming and Snowpipe complement each other. Use Snowpipe Streaming when data arrives as rows from applications, devices, or services and you need low-latency data availability. Use Snowpipe when your pipeline already produces files in cloud storage and batch-oriented, higher-latency loading is acceptable.