Tutorial: Get started with Snowpipe Streaming high performance architecture SDK¶

Generate a key pair for authentication¶

Generate a private-public key pair for authentication using OpenSSL. For more information, see Key-pair authentication and key-pair rotation.

Run the following commands in your terminal to generate the keys:

openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub

PUBK=$(cat ./rsa_key.pub | grep -v KEY- | tr -d '\012')
echo "ALTER USER MY_USER SET RSA_PUBLIC_KEY='$PUBK';"

Create database, schema, table, pipe, and configure user authentication¶

Run the following SQL commands in your Snowflake account (for example, using Snowsight or SnowSQL) to create the necessary objects and configure authentication for your user. Replace placeholders like MY_USER, MY_DATABASE, etc., with your desired names.

-- Create Database and Schema

CREATE OR REPLACE DATABASE MY_DATABASE;
CREATE OR REPLACE SCHEMA MY_SCHEMA;

-- Create Target Table

CREATE OR REPLACE TABLE MY_TABLE (
    data VARIANT,
    c1 NUMBER,
    c2 STRING
);

-- Create PIPE Object for Streaming Ingestion

CREATE OR REPLACE PIPE MY_PIPE
AS COPY INTO MY_TABLE FROM (SELECT $1, $1:c1, $1:ts FROM TABLE(DATA_SOURCE(TYPE => 'STREAMING')));

-- Configure Authentication Policy (optional, but recommended for explicit control)

CREATE OR REPLACE AUTHENTICATION POLICY testing_auth_policy
  AUTHENTICATION_METHODS = ('SAML', 'PASSWORD', 'OAUTH', 'KEYPAIR')
  CLIENT_TYPES = ('SNOWFLAKE_UI', 'SNOWSQL', 'DRIVERS');

-- Create or Modify User for SDK Authentication

CREATE OR REPLACE USER MY_USER; -- Or ALTER USER MY_USER if it already exists

-- Grant necessary roles to the user

GRANT ROLE ACCOUNTADMIN TO USER MY_USER; -- Grant a role with sufficient privileges

-- Apply authentication policy (if created)

ALTER USER MY_USER SET AUTHENTICATION POLICY testing_auth_policy;

-- Set default role (optional)

ALTER USER MY_USER SET DEFAULT_ROLE=ACCOUNTADMIN;

-- Set the public key for key-pair authentication

ALTER USER MY_USER SET RSA_PUBLIC_KEY='YOUR_FORMATTED_PUBLIC_KEY';

Create a profile configuration file¶

Create or update the profile.json file in the root directory of the provided Java demo project.

To format your private key (rsa_key.p8) for inclusion in the profile.json file, execute the following command in your terminal:

sed 's/$/\\n/' rsa_key.p8 | tr -d '\n'

This command will output your private key with newline characters escaped, making it suitable for a JSON string.

profile.json template¶

{
    "user": "MY_USER",
    "account": "your_account_identifier",
    "url": "https://your_account_identifier.snowflakecomputing.com:443",
    "private_key": "YOUR_FORMATTED_PRIVATE_KEY"
}

Replace the placeholders:

• MY_USER: Your Snowflake username configured in the previous step.

• your_account_identifier: Your Snowflake account identifier (for example, xy12345).

• YOUR_FORMATTED_PRIVATE_KEY: The output from the sed command executed above. Example format: -----BEGIN PRIVATE KEY-----\n-----END PRIVATE KEY-----.

Add the JAR dependency¶

To include the Snowpipe Streaming SDK, add the following dependency to your Maven pom.xml. Maven will automatically download the JAR from the public repository.

<dependency>
    <groupId>com.snowflake</groupId>
    <artifactId>snowpipe-streaming</artifactId>
    <version>YOUR_SDK_VERSION</version>
</dependency>
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.18.1</version>
</dependency>

Place the profile file¶

Ensure the profile.json file (configured in Step 2) is located in the root directory of your project.

Build and execute¶

• Navigate to the project’s root directory in your terminal.

• Build the project:

  mvn clean install
  

  Copy

• Execute the main class:

  mvn exec:java -Dexec.mainClass="com.snowflake.snowpipestreaming.demo.Main"
  

  Copy