Set up the Openflow Connector for Snowflake to Kafka¶

Prerequisites¶

1. Ensure that you have reviewed About Openflow Connector for Snowflake to Kafka.

2. Ensure that you have set up Openflow.

3. Create a Snowflake stream that will be queried for the changes.

4. Create a Kafka topic that will receive CDC messages from the Snowflake stream.

Set up Snowflake account¶

As a Snowflake account administrator, perform the following tasks:

1. Create the database, source table, and the stream object that the connector will use for reading CDC events. For example:

   create database stream_db;
   use database stream_db;
   create table stream_source (user_id varchar, data varchar);
   create stream stream_on_table on table stream_source;
   

   Copy

2. Create a new role or use an existing role, and grant the SELECT privilege on the stream and the source object for the stream. The connector will also need the USAGE privilege on the database and schema containing the stream and source object for the stream. For example:

   create role stream_reader;
   grant usage on database stream_db to role stream_reader;
   grant usage on schema stream_db.public to role stream_reader;
   grant select on stream_source to role stream_reader;
   grant select on stream_on_table to role stream_reader;
   

   Copy

3. Create a new Snowflake service user with the type as SERVICE. For example:

   create user stream_user type = service;
   

   Copy

4. Grant the Snowflake service user the role you created in the previous steps. For example:

   grant role stream_reader to user stream_user;
   

   Copy

5. Configure with key-pair auth for the Snowflake SERVICE user from step 3.

6. Snowflake strongly recommends this step. Configure a secrets manager supported by Openflow, for example, AWS, Azure, and Hashicorp, and store the public and private keys in the secret store. However, note that the private key generated in step 4 can be used directly as a configuration parameter for the connector configuration. In such a case, the private key is stored in Openflow runtime configuration.

   Note

   If for any reason, you do not wish to use a secrets manager, then you are responsible for safeguarding the public key and private key files used for key-pair authentication according to the security policies of your organization.

   1. Once the secrets manager is configured, determine how you will authenticate to it. On AWS, it’s recommended that you the EC2 instance role associated with Openflow as this way no other secrets have to be persisted.

   2. In Openflow, configure a Parameter Provider associated with this Secrets Manager, from the hamburger menu in the upper right. Navigate to Controller Settings » Parameter Provider and then fetch your parameter values.

   3. At this point all credentials can be referenced with the associated parameter paths and no sensitive values need to be persisted within Openflow.

7. Designate a warehouse for the connector to use. One connector can replicate single table to a single Kafka Topic. For this kind of processing, you can select the smallest warehouse.

Set up the connector¶

As a data engineer, perform the following tasks to install and configure a connector:

1. Navigate to the Openflow Overview page. In the Featured connectors section, select View more connectors.

2. On the Openflow connectors page, find and choose the connector depending on what kind of Kafka broker instance the connector should communicate with.

   • mTLS version: Choose this connector if you are using the SSL (mutual TLS) security protocol, or if you are using the SASL_SSL protocol and connecting to the broker that is using self-signed certificates.

   • SASL version: Choose this connector if you are using any other security protocol

3. Select Add to runtime.

4. In the Select runtime dialog, select your runtime from the Available runtimes drop-down list.

5. Select Add.

6. Authenticate to the deployment with your Snowflake account credentials and select Allow when prompted to allow the runtime application to access your Snowflake account. The connector installation process takes a few minutes to complete.

7. Authenticate to the runtime with your Snowflake account credentials.

   The Openflow canvas appears with the connector process group added to it.

8. Right-click on the imported process group and select Parameters.

9. Populate the required parameter values as described in Flow parameters.

Run the flow¶

1. Right-click on the plane and select Enable all Controller Services.

2. Right-click on the imported process group and select Start. The connector starts the data ingestion.