About Openflow Connector for MySQL¶
Note
The connector is subject to the Connector Terms.
This topic describes the basic concepts of Openflow Connector for MySQL, its workflow, and limitations.
The Openflow Connector for MySQL connects a MySQL database instance to Snowflake and replicates data from selected tables in near real-time or on a specified schedule. The connector also creates a log of all data changes, which is available along with the current state of the replicated tables.
How tables are replicated¶
The tables are replicated in the following stages:
Schema introspection: The connector discovers the columns in the source table, including the column names and types, then validates them against Snowflake’s and the connector’s Limitations. Validation failures cause this stage to fail, and the cycle completes. After successful completion of this stage, the connector creates an empty destination table.
Snapshot load: The connector copies all data available in the source table into the destination table. If this stage fails, then no more data is replicated. After successful completion, the data from the source table is available in the destination table.
Incremental load: The connector tracks changes in the source table and applies those changes to the destination table. This process continues until the table is removed from replication. Failure at this stage permanently stops replication of the source table, until the issue is resolved.
Note
Interim failures (such as connection errors) do not prevent the table from being replicated. However, permanent failures (such as unsupported data types) do prevent the table from being replicated.
If a permanent failure prevents a table from being replicated, remove the table from the list of tables to be replicated. After you address the problem that caused the failure, you can add the table back to the list of tables to be replicated.
Workflow¶
A MySQL database administrator performs the following tasks:
Configure MySQL replication settings
Create credentials for the connector
(Optionally) Provide the SSL certificate.
A Snowflake account administrator performs the following tasks:
Creates a service user for the connector, a warehouse for the connector, and a destination database for the replicated data.
Downloads and imports the connector definition file into the Snowflake Openflow canvas.
Specifies the required parameters for the flow template.
Runs the flow. The connector performs the following tasks when run in Openflow:
Creates a schema for journal tables.
Creates the schemas and destination tables matching the source tables configured for replication.
Starts replicating the tables. For details on the replication process, see How tables are replicated.
Supported MySQL versions¶
The following table lists the tested and officially supported MySQL versions.
8.0 |
8.4 |
|
|---|---|---|
Yes |
Yes |
|
Yes |
||
Yes, as Version 3 |
||
Yes |
Yes |
|
No |
Limitations¶
The connector supports MySQL version 8 or later.
The connector supports only username and password authentication with MySQL.
The connector does not replicate tables with data that exceeds Snowflake’s type limitations.
The connector does not replicate columns of types GEOMETRY, GEOMETRYCOLLECTION, LINESTRING, MULTILINESTRING, MULTIPOINT, MULTIPOLYGON, POINT, and POLYGON.
The connector has the Group Replication Limitations of MySQL. This means that a single transaction must fit into a binary log message of size no more than 4 GB.
The connector requires every replicated table to have a primary key.
The connector supports source table schema changes with the exception of changing primary key definitions and changing the precision or the scale of a numeric column.
Note
Limitations affecting certain table columns can be bypassed by excluding these specific columns from replication.