Elastic Channels¶

Elastic Channels is a feature for Snowpipe Streaming, available on the latest high-performance architecture, that lets you stream data into Snowflake without managing channel state. You write rows directly to a single, implicitly created ELASTIC channel on your pipe, and Snowflake handles scaling and durable acknowledgment for you. Many concurrent clients can write to the same ELASTIC channel without coordinating with each other, which simplifies fan-in use cases where multiple data producers stream into the same Snowflake table. Elastic Channels are designed for high-throughput, stateless ingestion that doesn’t require strict exactly-once delivery.

Supported deployments¶

During private preview, Elastic Channels are available on AWS Commercial deployments only.

What problem Elastic Channels solve¶

Standard Snowpipe Streaming channels are named, ordered, and exactly-once. To get those guarantees, your client manages sequencers and offset tokens, and to scale beyond the per-channel throughput limit you have to shard your data across many channels and track each one.

For a large class of workloads, relaxed delivery semantics greatly simplify the developer experience:

• IoT and telemetry fleets with tens of thousands of devices, where each handshake to open and track a channel is more expensive than the data the device produces.
• Stateless agents like SQS or Pub/Sub readers and serverless workers, where the agent has no durable place to store an offset token between runs.
• Non-rewindable data sources like live sensor feeds, network packet streams, or live event broadcasts, where the source itself can’t replay messages you’ve missed. Strict exactly-once tracking adds complexity without benefit when lost messages can’t be recovered upstream anyway.
• Massive-scale analytics pipelines that need tens of GB/s into a single logical endpoint without building a custom sharding layer on top of Snowpipe Streaming.

For these workloads, Elastic Channels removes the need to manage channels yourself. You don’t open or close channels, you don’t pick channel names, and you don’t track offset tokens. You just append rows, and a success response implies the row is now durably persisted in Snowflake.

When to use Elastic Channels¶

Use Elastic Channels when:

• You can tolerate at-least-once delivery (rare duplicates are acceptable, or your downstream is idempotent).
• You don’t need strict per-channel ordering across all rows.
• You want a single ingestion endpoint that scales horizontally on the server side without client-side sharding.
• Your producer is stateless or short-lived and can’t maintain offset tokens between restarts.

Use standard named channels instead when you need:

• Exactly-once delivery with replay protection.
• Strict ordering of rows on a single logical stream.
• Per-channel offset tracking integrated with an external source (for example, Kafka partition offsets).

Durable acknowledgments¶

Every appendRow and appendRows call returns a durable acknowledgment that completes only after Snowflake has durably persisted the data. With the SDK, that acknowledgment is a future returned by the call; with the REST API, it’s the HTTP response on the insert-rows endpoint. Once the acknowledgment resolves successfully, the rows are guaranteed to be ingested into the target table, or routed to the configured error table if the row shape doesn’t match the target schema.

The durable acknowledgment makes the API effectively synchronous when you want it to be. A producer can:

1. Build a row.
2. Call appendRow(row) and await the returned future (SDK) or send the request and read the HTTP response (REST).
3. Discard its local copy of the row as soon as the acknowledgment completes.

Because durability is established server-side before the acknowledgment completes, stateless clients don’t need to keep a local copy of unacknowledged rows. If the acknowledgment doesn’t complete (because of a network failure, client crash, or server error), the producer hasn’t been told the data is safe and should resend. If the row was actually persisted before the failure, the resend produces a duplicate, which is the at-least-once trade-off.

If the row schema doesn’t match the target table, the acknowledgment completes successfully because the row is durably persisted to the error table. Always check the error table for rejected rows.

Comparing standard channels and Elastic Channels¶

The following table summarizes the differences:

How Elastic Channels work¶

Each streaming pipe automatically exposes a single logical channel named ELASTIC. You don’t create, open, or drop it. The first time your client requests the elastic channel for a pipe, the SDK begins routing rows to it.

Snowflake scales the elastic channel automatically to sustain very high throughput, without you splitting data across multiple channels. Ordering isn’t preserved across the channel because rows are distributed server-side. Continuation tokens and offset tokens don’t apply to elastic channels and aren’t part of the API.

The name ELASTIC is reserved. You can’t use it with the standard channel APIs:

• Calling openChannel("ELASTIC") (or any case variant) on a standard channel API returns ElasticChannelReservedName.
• Calling dropChannel("ELASTIC") returns CannotDropElasticChannel. The elastic channel’s lifecycle is tied to the client, so you don’t need to drop it.

Required privileges and authentication¶

Elastic Channels use the same privilege model and authentication mechanisms as standard Snowpipe Streaming on the high-performance architecture. The privilege grants differ slightly between default pipes (auto-created by writing to a table) and custom pipes; for the full grant list and the difference between the two, see The PIPE object.

Both key pair (JWT) and Programmatic Access Token (PAT) authentication are supported. JWT is the default; you only need to set authorization_type explicitly when using a PAT. For setup, see Tutorial: Get started with Snowpipe Streaming high-performance architecture SDK, Key-pair authentication and key-pair rotation, and Using programmatic access tokens for authentication.

Get started¶

Step 1: Create the target table¶

Elastic Channels write to a streaming pipe whose name Snowflake derives by appending -STREAMING to the target table name. You don’t need to create the pipe explicitly.

Create a table for your streaming data:

CREATE OR REPLACE TABLE sensor_readings (
  device_id   INT,
  reading_ts  TIMESTAMP_NTZ,
  temperature NUMBER(6, 2)
);

When the SDK first writes to this table, Snowflake creates a default streaming pipe named SENSOR_READINGS-STREAMING automatically.

Step 2: Configure your profile¶

If you’ve already set up Snowpipe Streaming for this account, you can reuse the same profile.json. Snowpipe Streaming defaults to key pair (JWT) authentication, so you don’t need to set authorization_type explicitly unless you’re using a PAT. The following minimal example shows the table-mode-relevant fields for the default key pair authentication:

{
  "account": "<account_identifier>",
  "url": "https://<account_identifier>.snowflakecomputing.com:443",
  "user": "MY_USER",
  "private_key_file": "rsa_key.p8",
  "role": "MY_ROLE"
}

To use a PAT instead, set authorization_type to PAT and replace the key pair fields with the token:

{
  "authorization_type": "PAT",
  "account": "<account_identifier>",
  "url": "https://<account_identifier>.snowflakecomputing.com:443",
  "token": "<your_programmatic_access_token>",
  "role": "MY_ROLE"
}

For the full schema and field reference, see Step 2: Configure an authentication profile.

Step 3: Install the SDK¶

Elastic Channels require version 1.6.2 or later of the Snowpipe Streaming SDK. For full installation details and supported platforms, see Tutorial: Get started with Snowpipe Streaming high-performance architecture SDK.

python3 -m venv .venv
source .venv/bin/activate
pip install "snowpipe-streaming>=1.6.2"

Step 4: Append rows to the elastic channel¶

The producer code differs from standard-channel producer code in three ways:

• Build the client with the table-mode factory (tableBuilder in Java, from_table in Python). This factory derives the pipe name from the table name automatically.
• Get the elastic channel by calling getElasticChannel() (Java) or get_elastic_channel() (Python). The same instance is returned on every call.
• Append rows and await the returned future. The future resolves only after the data is durably persisted, so you can drop your local copy of the row as soon as it completes.

from datetime import datetime, timezone
from snowflake.ingest.streaming import StreamingIngestClient

client = StreamingIngestClient.from_table(
    client_name="sensor_demo_client",
    db_name="MY_DB",
    schema_name="MY_SCHEMA",
    table_name="SENSOR_READINGS",
    profile_json="profile.json",
)

channel = client.get_elastic_channel()

row = {
    "DEVICE_ID": 1,
    "READING_TS": datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S.%f"),
    "TEMPERATURE": 68.5,
}

future = channel.append_row(row)
future.result()  # Blocks until Snowflake durably persists the row.

client.close()

For a fire-and-forget pattern, capture the futures and check them later (or in batches) instead of awaiting each one inline. See Working with durable-acknowledgment futures for guidance on draining futures asynchronously.

Working with durable-acknowledgment futures¶

Awaiting every future inline serializes ingestion to one round-trip per row. For higher throughput, fire many appendRow calls and drain the futures as a group:

from concurrent.futures import wait

futures = [channel.append_row(row) for row in rows]
wait(futures)

If any future fails, the corresponding rows weren’t ingested and should be retried by the producer.

Direct REST usage¶

You can also write to an elastic channel directly through the REST API instead of using the SDK. Send rows to the same insert-rows endpoint as a standard channel, but use the reserved channel name ELASTIC and omit the offsetToken and continuationToken query parameters:

POST {subdomain}/v2/streaming/data/databases/{db}/schemas/{schema}/pipes/{pipe}/channels/ELASTIC/rows

Both key pair JWT and PAT bearer tokens are accepted on the elastic-channel endpoint. For the full REST contract, including the bearer token format and request payload schema, see Snowpipe Streaming REST API endpoints.

The REST contract for elastic channels may change before public preview. The SDK is the recommended client for private preview.

Limitations¶

• During private preview, only the Java and Python SDKs expose elastic-channel APIs. Node.js, Go, and Rust support is planned for a later preview update.
• Elastic Channels provide at-least-once delivery only. If your workload requires exactly-once semantics, use standard named channels.
• Per-row ordering isn’t preserved across the elastic channel. If your downstream depends on ordering, use a column on the row (such as an event timestamp) and sort at query time.
• The ELASTIC channel name is reserved. The SDK rejects openChannel("ELASTIC") and dropChannel("ELASTIC") on the standard channel APIs.
• The elastic channel doesn’t expose close(), waitForCommit, or any offset-token methods. Wait on the per-call futures returned by appendRow and appendRows instead.