Openflow Connector for Amazon Kinesis Data Streams¶
Bemerkung
Dieser Connector unterliegt den `Nutzungsbedingungen für Snowflake Connector<https://www.snowflake.com/legal/snowflake-connector-terms/>`_.
About¶
In diesem Thema werden die grundlegenden Konzepte, der Workflow und die Einschränkungen von Openflow Connector for Amazon Kinesis Data Streams beschrieben.
Sie können Amazon Kinesis Data Streams verwenden, um große Streams von Datensätzen in Echtzeit zu sammeln und zu verarbeiten. Produzenten senden kontinuierlich Daten an Kinesis Data Streams, und Verbraucher verarbeiten die Daten in Echtzeit.
Ein Kinesis-Datenstream ist ein Satz von Shards. Jeder Shard enthält eine Sequenz von Datensätzen. Ein Datensatz ist die Einheit der in einem Kinesis-Datenstream gespeicherten Daten. Datensätze bestehen aus einer Sequenznummer, einem Partitionsschlüssel und einem Daten-Blob, also einer unveränderlichen Folge von Bytes.
Openflow Connector for Amazon Kinesis Data Streams reads data from Kinesis streams and writes it into Snowflake tables using the Snowpipe Streaming architecture.
Use this connector if you’re looking to do the following:
Ingest real-time events from Amazon Kinesis into Snowflake for near real-time analytics
Ingest real-time events from Amazon Kinesis into Snowflake-managed Iceberg™ tables
Accelerate your ingestion even more by combining Openflow speed with the Interactive Tables feature
Use Single Message Transforms to enrich or filter data before it lands in Snowflake.
Einschränkungen¶
One connector supports only ingestion from a single stream.
The connector does not support schema evolution for Apache Iceberg™ tables.
Autoscaling is not supported. The number of Openflow runtime min and max nodes should be constant for the runtime where Openflow Connector for Amazon Kinesis Data Streams is deployed.
Limitations of fault tolerance with the connector¶
Kinesis Streams can be configured with a retention time. If for any reason the Openflow Connector for Amazon Kinesis Data Streams is not able to ingest data for more than the retention time, then expired records will not be loaded.
Supported data types and authentication methods¶
The connector by default is configured to work with the JSON data type and supports authentication using AWS Credentials: Access Key ID and Secret Access Key. Connector can be customized to work with other data types and authentication methods.