About Openflow Connector for Kinesis Data Streams

Note

This connector is subject to the Snowflake Connector Terms.

This topic describes the basic concepts of Openflow Connector for Kinesis Data Streams, its workflow, and limitations.

You can use Amazon Kinesis Data Streams to collect and process large streams of data records in real time. Producers continually push data to Kinesis Data Streams, and consumers process the data in real time.

A Kinesis data stream is a set of shards. Each shard has a sequence of data records. A data record is the unit of data stored in a Kinesis data stream. Data records are composed of a sequence number, a partition key, and a data blob, which is an immutable sequence of bytes.

The Openflow Connector for Kinesis Data Streams reads data from a Kinesis data stream and writes it to a Snowflake table using Snowpipe Streaming.

Use this connector if you’re looking to do the following:

  • Ingest real‐time events from Amazon Kinesis Data Streams into Snowflake for near real-time analytics

Workflow

  1. An AWS administrator performs the following tasks:

    1. Creates credentials for the connector to connect with Kinesis Stream and the associated DynamoDB.

    2. Sets up IAM policies that have the permissions listed in IAM permissions required for KCL consumer applications.

    3. Record the stream name and application name and give them to your Snowflake account administrator. These are required when setting up the connector in Runtime.

  2. A Snowflake account administrator performs the following tasks:

    1. Installs the connector.

    2. Configures the connector as follows:

      1. Provides the AWS and Snowflake credentials and settings.

      2. Provides the Kinesis stream name.

      3. Sets the database and schema names in the Snowflake account.

      4. Customizes other parameters.

    3. Runs the connector in the Openflow canvas. Upon execution, the connector performs the following actions:

      1. Creates DynamoDB tables for storing Kinesis Stream checkpoints.

      2. Extracts stream data.

      3. Creates the configured destination table in the Snowflake database if at least one record was received from the stream.

      4. Loads the processed data into the specified Snowflake table.

  3. Business users can perform operations on the data downloaded from Kinesis into the destination table.

Limitations

  • Only a single stream is supported.

  • Enhanced fan-out mode is always enabled.

  • If a manually created table is used:

    • A table name must precisely match the stream of the data it holds.

    • A table name must be in uppercase format.

  • Only JSON message format is supported.

  • Only Amazon Access Key IAM authentication is supported.

  • Failed messages are logged to the Snowflake logs and not routed to a DLQ stream.

Next steps

Set up Openflow Connector for Kinesis Data Streams for JSON data format

Language: English