Openflow Connector for PostgreSQL: Set up incremental replication without snapshots¶

Note

This connector is subject to the Snowflake Connector Terms.

The Openflow Connector for PostgreSQL connector can be configured to immediately start replicating incremental changes for newly added tables, bypassing snapshots. Incremental load is useful when reinstalling the connector over previously replicated data and to continue replication without snapshotting every table again.

Incremental replication can be enabled in a new instance of the connector or in an existing one.

To enable incremental replication in a new instance of the connector perform the following tasks:

  1. Setup the connector as described in Set up the Openflow Connector for PostgreSQL.

  2. In the PostgreSQL Ingestion Parameters context, set the Ingestion Type parameter to incremental.

Enable incremental replication without snapshots¶

To enable incremental replication on an existing connector:

  1. Sign in to Snowsight.

  2. In the navigation menu, select Ingestion » Openflow.

  3. In the Openflow pane select the Runtimes tab.

  4. Select the runtime containing the connector.

  5. Select the connector.

  6. In the Ingestion Parameters context, specify Ingestion Type = incremental.

  7. Add new replication tables. These tables immediately switch to their incremental load.

Note

To return to replicating tables with the snapshot load, change Ingestion Type from incremental to full.

Usage notes¶

  • Changing the value of Ingestion Type does not impact any tables that have begun replicating data. Tables currently in the snapshot phase continue until the snapshot load is complete.

  • While Ingestion Type is set to incremental, new tables added to the list of replicated tables bypass the snapshot phase. This includes new tables added to the source database that match the Included Table Regex parameter. Ensure that the ingestion type is set to incremental to bypass the snapshot phase.

    Note

    Connectors should only remain in incremental mode as long as required as it bypasses snapshots. Once customer needs for incremental updates have been satisfied the connector should be returned to full mode.

  • For tables that bypass snapshot load, the connector creates a destination table in Snowflake, by executing CREATE TABLE IF NOT EXISTS, only if no destination table already exists. Tables going through the snapshot require that no destination table exist.

Language: English