Openflow Connector for Amazon Kinesis Data Streams の維持

注釈

このコネクタは、 Snowflakeコネクタ規約 に従うものとします。

このトピックでは、状態の管理とリセットの方法を含む Openflow Connector for Amazon Kinesis Data Streams コネクタを維持する方法について説明します。

コネクタの状態を管理する

The Openflow Connector for Amazon Kinesis Data Streams connector uses DynamoDB to store the consumer application state.

DynamoDB table created by the connector

The connector creates a DynamoDB table with the name specified in AWS Kinesis Application Name. The table stores the checkpointed sequence number for each shard in the stream. This tracks which records have been processed.

If multiple processors use the same application name, they cooperate to consume data from the stream and share this table. If processors have different application names, each creates its own table to independently track consumed records.

コネクタの状態をリセットする

DynamoDBのコネクタの状態が破損するか、一貫性が失われた場合は、リセットする必要があります。コネクタの状態をリセットするには、2つのアプローチがあります。

Reset by changing the application name

The simplest way to reset the connector state is to change the AWS Kinesis Application Name parameter:

  1. コネクタを停止します。

  2. コネクタのパラメーターコンテキストに移動します。

  3. Change the AWS Kinesis Application Name parameter value to a new value.

  4. コネクタを起動します。

The connector creates a new DynamoDB table with the new application name and begins consuming records from the position specified by the AWS Kinesis Initial Stream Position parameter.

注釈

  • When you change the application name, the connector doesn't delete the old DynamoDB table. You must manually delete it through the AWS Console or the AWS CLI.

  • If your IAM policy restricts DynamoDB access to a specific table name, you must update the policy to allow access to the new table name. For more information on configuring IAM permissions, see Openflow Connector for Amazon Kinesis Data Streams を設定する.

Reset by deleting the DynamoDB table

Alternatively, you can delete the existing DynamoDB table to reset the state:

  1. コネクタを停止します。

  2. In the AWS Console or using the AWS CLI, delete the DynamoDB table associated with the application name.

  3. コネクタを起動します。

The connector recreates the table and begins consuming records from the position specified by the AWS Kinesis Initial Stream Position parameter.

警告

Resetting the connector state causes the connector to reprocess records from the position specified by the initial stream position. Depending on your AWS Kinesis Initial Stream Position setting, this may result in duplicate data being ingested into Snowflake or data not being ingested at all.