Set up the Openflow Connector for Snowflake to Kafka¶

This topic describes the steps to set up the Openflow Connector for Snowflake to Kafka.

Prerequisites¶

1. Ensure that you have reviewed About Openflow Connector for Snowflake to Kafka.
2. Ensure that you have Set up Openflow - Snowflake Deployments or Set up Openflow - BYOC.
3. Create a Snowflake stream that will be queried for the changes.
4. Create a Kafka topic that will receive CDC messages from the Snowflake stream.

Set up Snowflake account¶

As a Snowflake account administrator, perform the following tasks. With the default SNOWFLAKE_MANAGED authentication strategy, the runtime’s execute-as role is the identity the connector uses to access Snowflake, so you grant it the privileges below.

1. Create the database, source table, and the stream object that the connector will use for reading CDC events. For example:

   Copy code

   create database stream_db;
   use database stream_db;
   create table stream_source (user_id varchar, data varchar);
   create stream stream_on_table on table stream_source;
   

2. Grant the runtime’s execute-as role the SELECT privilege on the stream and the source object for the stream, and the USAGE privilege on the database and schema containing them. For example:

   Copy code

   grant usage on database stream_db to role OPENFLOW_<RUNTIME_NAME>_EXECUTE_AS_RL;
   grant usage on schema stream_db.public to role OPENFLOW_<RUNTIME_NAME>_EXECUTE_AS_RL;
   grant select on stream_source to role OPENFLOW_<RUNTIME_NAME>_EXECUTE_AS_RL;
   grant select on stream_on_table to role OPENFLOW_<RUNTIME_NAME>_EXECUTE_AS_RL;
   

3. Designate a warehouse for the connector to use. One connector can replicate a single table to a single Kafka topic. For this kind of processing, you can select the smallest warehouse.

   Copy code

   create warehouse if not exists <openflow_warehouse>
     with
     warehouse_size = 'XSMALL'
     auto_suspend = 300
     auto_resume = true;
   
   grant usage, operate on warehouse <openflow_warehouse> to role OPENFLOW_<RUNTIME_NAME>_EXECUTE_AS_RL;
   

Set up the connector¶

As a data engineer, perform the following tasks to install and configure a connector:

1. Navigate to the Connector library tab in Openflow.

2. On the Openflow connectors page, find and choose the connector depending on what kind of Kafka broker instance the connector should communicate with.

   • mTLS version: Choose this connector if you are using the SSL (mutual TLS) security protocol, or if you are using the SASL_SSL protocol and connecting to the broker that is using self-signed certificates.
   • SASL version: Choose this connector if you are using any other security protocol.

3. Select Install.

4. In the Select runtime dialog, select your runtime from the Available runtimes drop-down list.

5. Select Add.

6. Authenticate to the deployment with your Snowflake account credentials and select Allow when prompted to allow the runtime application to access your Snowflake account. The connector installation process takes a few minutes to complete.

7. Authenticate to the runtime with your Snowflake account credentials.

   The Openflow canvas appears with the connector process group added to it.

8. Right-click on the imported process group and select Parameters.

9. Populate the required parameter values as described in Flow parameters.

Flow parameters¶

This section describes the flow parameters that you can configure based on the following parameter contexts:

• Kafka Sink Source Parameters
• Kafka Sink Destination Parameters
• Kafka Sink Ingestion Parameters

Kafka Sink Source Parameters¶

Kafka Sink Destination Parameters¶

Kafka Sink Ingestion Parameters¶

Run the flow¶

1. Right-click on the canvas and select Enable all Controller Services.
2. Right-click on the imported process group and select Start. The connector starts the data ingestion.