Tutorial: Get started with Named Channels using the SDK¶
This tutorial provides step-by-step instructions for setting up and running a Named Channel demo application with the snowpipe-streaming SDK. Named Channels provide ordered, exactly-once ingestion. Exactly-once recovery requires records retained in the source or durable application-managed storage for replay.
For Elastic Channels (the recommended starting point for most new applications), see Tutorial: Get started with Elastic Channels (SDK).
Prerequisites¶
Before you run the demo, ensure that you meet the following prerequisites:
-
Snowflake account: Verify that you have access to a Snowflake account. You will need a user with sufficient privileges (e.g., ACCOUNTADMIN or USERADMIN for the initial setup) to create the dedicated user and custom role detailed in Step 1: Configure Snowflake objects.
-
Network access: Complete the network connectivity prerequisites from your producer runtime. Verify the account endpoint, discovered ingest endpoint, and cloud-storage upload destinations. Private connectivity also requires DNS configuration for the ingest hostname.
-
Java Development Environment: Install Java 11 or later, and Maven for dependency management.
-
Python: Install Python version 3.9 or later.
-
C++: Install a C++ compiler that supports C++17 or later.
-
Node.js: Install Node.js version 20 or later.
-
System requirements: The SDK requires glibc version 2.26 or later. You can check your current glibc version with:
-
Snowpipe Streaming SDKs and the sample code:
Download the sample code for your preferred language from the Snowpipe Streaming SDK examples in the GitHub repository.
Prerequisites and setup¶
The following steps configure the Snowflake objects, authentication, and SDK dependency required for Named Channel ingestion.
Step 1: Configure Snowflake objects¶
Before you can use the snowpipe-streaming SDK, you must create a target table within your Snowflake environment. Unlike the classic architecture, the high-performance architecture requires a PIPE object for data ingestion. This tutorial uses the default pipe that is automatically created at ingest time for your target table. If you require additional features, such as in-flight transformations or clustering at ingest time, see CREATE PIPE.
Generate a key pair for authentication¶
Generate a private-public key pair for authentication using OpenSSL. For more information, see Key-pair authentication and key-pair rotation.
Run the following commands in your terminal to generate the keys:
Important
Save the generated rsa_key.p8 (private key) and rsa_key.pub (public key) files securely. You will use these keys in subsequent authentication steps.
Create database, schema, table, and configure user authentication¶
Run the following SQL commands in your Snowflake account; for example, by using Snowsight or Snowflake CLI). You must have a role with permissions to create users, roles, and databases — such as ACCOUNTADMIN or USERADMIN for the first few lines, and then switching to the new role. Replace placeholders like MY_USER, MY_ROLE, MY_DATABASE, and so on, with the names that you want.
Note
The data column in the sample table is a VARIANT, so pass its value as an object, not a JSON-encoded string, which is stored as a plain string. In Java, Python, and Node.js, use a native object (Java Map, Python dict, or JavaScript object). In C++, where each row is a JSON string, write data as a nested JSON object, for example {"DATA": {"event_id": 1}}.
Step 2: Configure an authentication profile¶
The demo application requires a profile.json file to store connection settings, including authentication details. The SDK uses key-pair authentication for secure connections.
Create a profile configuration file¶
Create or update the profile.json file in the root directory of your demo project.
profile.json template¶
Replace the placeholders:
MY_USER: Your Snowflake username configured in Step 1: Configure Snowflake objects.your_account_identifier: Your Snowflake account identifier (for example,xy12345).rsa_key.p8: The private key file you generated in Step 1: Configure Snowflake objects.MY_ROLE: The dedicated role (MY_ROLE) you created and granted to the user in Step 1: Configure Snowflake objects.
Step 3: Set up the demo project¶
Download: Sample Java code
Add the JAR dependency
To include the Snowpipe Streaming SDK, add the following dependency to your Maven pom.xml. Maven automatically downloads the JAR from the public repository.
Important
Replace YOUR_SDK_VERSION with the specific version available on Maven Central.
Download: Sample Python code
Add the Python dependency
The SDK requires Python version 3.9 or later.
To install the Snowpipe Streaming SDK for Python, run the following command:
For more information about the package, see PyPI.
Download: Sample C++ code
Add the C++ dependency
The SDK requires C++ version 17 or later.
Download the C++ Snowpipe Streaming SDK from the Snowflake Client Repository.
Each package in the repository has a matching .sha256 checksum file and .asc signature file. The following commands download all three files into the current directory, then validate and unpack the package. Set SDK_VERSION to the version you want, PLATFORM to your package’s platform, and PLATFORM_DIR to the matching repository directory:
PLATFORM | PLATFORM_DIR |
|---|---|
Linux-x86_64 | linux_x86_64 |
Linux-aarch64 | linux_aarch64 |
Darwin-arm64 | macos_arm64 |
Set GPG_KEY_ID to the ID of the Snowflake public key that signed the package. Beginning with version 1.9.4, packages are signed with the Snowflake public key 3C98F63C9292CE02.
Do not use the package if either check fails.
Note
Verifying the signature produces a warning similar to the following:
To avoid the warning, you can grant the Snowflake GPG public key implicit trust.
Important
Keep the unpacked package directory intact when you build. The package’s build files (the CMake package configuration file and the Bazel BUILD file) locate the headers and libraries by their paths inside the package, so move or rename the whole directory rather than copying individual files out of it.
Add the SDK to your CMakeLists.txt. The following file builds example.cc into an example executable that links to the SDK. When you build, set CMAKE_PREFIX_PATH to the absolute path of the unpacked package directory (SDK_DIR from the previous step).
To build your application, configure CMake with the SDK package directory, then build:
Download: Sample Node.js code
Add the Node.js dependency
The SDK requires Node.js version 20 or later.
To install the Snowpipe Streaming SDK for Node.js, run the following command:
For more information about the package, see npm.
Place the profile file¶
Ensure that the profile.json file that you configured in Step 2: Configure an authentication profile is located in the root directory of your project.
Open and use a Named Channel¶
Use a Named Channel when your application requires ordering within a channel or exactly-once recovery. Complete Steps 1 through 3 in Prerequisites and setup before running the following code.
The examples open MY_CHANNEL with an initial offset token, asynchronously append three rows with offset tokens 1, 2, and 3 without waiting on each individual append, retrieve channel status, then wait once for offset token 3 to commit before closing.
Append rows as they arrive; the SDK combines appends internally using time and size thresholds. Submit rows serially in source order within each Named Channel, but don’t wait for every row to commit. Bound outstanding work and checkpoint periodically without collecting rows into another batch before submitting them.
Important
Don’t call waitForCommit after every row or small batch. Waiting on every append serializes ingestion around commit latency and defeats the purpose of asynchronous appends. waitForCommit polls channel status until the target offset commits, so reserve it for checkpoints, source handoffs, or graceful shutdown.
See the full Java, Python, C++, and Node.js samples for complete examples.
Run the application, then query the target table to verify the rows. In a production application, retrieve the latest committed offset when you open the Named Channel and resume the source from the next record. For more information, see Offset tokens and exactly-once delivery.