Replicating Account Objects¶

Step 1. Disable Replication for a Replication Enabled Database¶

Execute the SYSTEM$DISABLE_DATABASE_REPLICATION function to disable replication for a primary database, along with any secondary databases linked to it, in order to add it to a replication or failover group.

Execute the following SQL statement from the source account with the primary database:

SELECT SYSTEM$DISABLE_DATABASE_REPLICATION('mydb');

Step 2. Add the Database to a Primary Failover Group and Create a Secondary Failover Group¶

Once you have successfully disabled replication for a database, you can add the primary database to a failover group in the source account.

Then create a secondary failover group in the target account. When the secondary failover group is refreshed in the target account, the previously secondary database will automatically be added as a member of the secondary failover group and refreshed with the changes from the primary database.

For more details on creating primary and secondary failover groups, see Workflow.

Examples¶

Execute the following SQL statements in your preferred Snowflake client to enable account and database object replication and failover, and refresh objects.

Prerequisite: Enable Replication for Accounts in the Organization¶

The organization administrator (ORGADMIN role) must enable replication for the source and target accounts.

To enable replication for accounts, a user with the ORGADMIN role uses the SYSTEM$GLOBAL_ACCOUNT_SET_PARAMETER function to set the ENABLE_ACCOUNT_DATABASE_REPLICATION parameter to true. Note that multiple accounts in an organization can be enabled for replication from the same ORGADMIN account.

Log into an ORGADMIN account to enable replication for each source and target account in your organization.

USE ROLE ORGADMIN;

-- View the list of the accounts in your organization
-- Note the organization name and account name for each account for which you are enabling replication
SHOW ORGANIZATION ACCOUNTS;

-- Enable replication by executing this statement for each source and target account in your organization
SELECT SYSTEM$GLOBAL_ACCOUNT_SET_PARAMETER('<organization_name>.<account_name>', 'ENABLE_ACCOUNT_DATABASE_REPLICATION', 'true');

Though the SYSTEM$GLOBAL_ACCOUNT_SET_PARAMETER function supports the legacy account locator identifier, it causes unexpected results when an organization has multiple accounts that share the same locator (in different regions).

Step 1: Create a Role with the CREATE FAILOVER GROUP Privilege in the Source Account — Optional¶

Create a role and grant it the CREATE FAILOVER GROUP privilege. This step is optional. If you have already created this role, skip to Step 2: Create a Primary Failover Group in a Source Account.

USE ROLE ACCOUNTADMIN;

CREATE ROLE myrole;

GRANT CREATE FAILOVER GROUP ON ACCOUNT
    TO ROLE myrole;

Step 2: Create a Primary Failover Group in a Source Account¶

Create a primary failover group and enable the replication and failover of specific objects from the current (source) account to one or more target accounts in the same organization.

SHOW REPLICATION ACCOUNTS;

+------------------+-------------------------------+--------------+-----------------+-----------------+-------------------+--------------+
| snowflake_region | created_on                    | account_name | account_locator | comment         | organization_name | is_org_admin |
+------------------+-------------------------------+--------------+-----------------+-----------------+-------------------+--------------+
| AWS_US_WEST_2    | 2020-07-15 21:59:25.455 -0800 | myaccount1   | myacctlocator1  |                 | myorg             | true         |
+------------------+-------------------------------+--------------+-----------------+-----------------+-------------------+--------------+
| AWS_US_EAST_1    | 2020-07-23 14:12:23.573 -0800 | myaccount2   | myacctlocator2  |                 | myorg             | false        |
+------------------+-------------------------------+--------------+-----------------+-----------------+-------------------+--------------+
| AWS_US_EAST_2    | 2020-07-25 19:25:04.412 -0800 | myaccount3   | myacctlocator3  |                 | myorg             | false        |
+------------------+-------------------------------+--------------+-----------------+-----------------+-------------------+--------------+

USE ROLE myrole;

CREATE FAILOVER GROUP myfg
    OBJECT_TYPES = USERS, ROLES, WAREHOUSES, RESOURCE MONITORS, DATABASES, INTEGRATIONS, NETWORK POLICIES
    ALLOWED_DATABASES = db1, db2
    ALLOWED_INTEGRATION_TYPES = API INTEGRATIONS
    ALLOWED_ACCOUNTS = myorg.myaccount2
    REPLICATION_SCHEDULE = '10 MINUTE';

Step 3: Create a Role with the CREATE FAILOVER GROUP Privilege in the Target Account — Optional¶

Create a role in the target account and grant it the CREATE FAILOVER GROUP privilege. This step is optional. If you have already created this role, skip to Step 4: Create a Secondary Failover Group in the Target Account.

USE ROLE ACCOUNTADMIN;

CREATE ROLE myrole;

GRANT CREATE FAILOVER GROUP ON ACCOUNT
    TO ROLE myrole;

Step 4: Create a Secondary Failover Group in the Target Account¶

Create a secondary failover group in the target account as a replica of the primary failover group in the source account.

Execute a CREATE FAILOVER GROUP … AS REPLICA OF statement in each target account for which you enabled replication in Step 2: Create a Primary Failover Group in a Source Account (in this topic).

Executed from each target account:

USE ROLE myrole;

CREATE FAILOVER GROUP myfg
  AS REPLICA OF myorg.myaccount1.myfg;

Step 5. Refresh a Secondary Failover Group in the Target Account Manually — Optional¶

To manually refresh the objects in a target account, execute the ALTER FAILOVER GROUP … REFRESH command.

As a best practice, we recommend scheduling your secondary refreshes by setting the REPLICATION_SCHEDULE parameter using CREATE FAILOVER GROUP or ALTER FAILOVER GROUP.

Use SYSTEM$LINK_ACCOUNT_OBJECTS_BY_NAME() to Apply Global IDs¶

You can prevent the loss of some object types by linking matching objects with the same name in the source and target accounts. The SYSTEM$LINK_ACCOUNT_OBJECTS_BY_NAME function adds a global identifier to account objects in the target account.

Apply global identifiers to account objects in the target account of the types included in the object_types list for failover group myfg:

Execute the following SQL statement using the ACCOUNTADMIN role:

SELECT SYSTEM$LINK_ACCOUNT_OBJECTS_BY_NAME('myfg');

Initial Replication of Users and Roles¶

The behavior of the initial refresh operation for USERS and ROLES object types can vary depending on whether or not there are matching objects with the same name in the target account.

• If there are existing users in the target account with the same name as users in the source account, the initial refresh operation fails and describes the two options you have to continue:

  • Force the refresh operation and allow any existing users in the target account to be dropped. The users in the source account will be replicated to the target account.

    To force a refresh for a group, use the FORCE parameter for the refresh command. For example, to force the refresh of a failover group, execute the following command:

    ALTER FAILOVER GROUP <fg_name> REFRESH FORCE;
    

    Copy

  • Link the account objects by name. The SYSTEM$LINK_ACCOUNT_OBJECTS_BY_NAME function links users with the same name in both the target account and the source account. Users in the target account that are linked are not deleted.

    To link account objects by name, execute the following command:

    SELECT SYSTEM$LINK_ACCOUNT_OBJECTS_BY_NAME('<rg_name>');
    

    Copy

    Note

    Any user in the target account that does not have a matching user in the source account with the same name is dropped.

• If there are no users in the target account with names matching users in the source account, the initial refresh operation in the target account drops all users. This can result in the following data and metadata loss:

  • If USERS are included in the OBJECT_TYPES list for a replication or failover group:

    • Worksheets are lost.

    • Query history is lost.

  • If USERS are included in the OBJECT_TYPES list, but ROLES is not:

    • Privilege grants to users are lost.

  • If ROLES are included in the OBJECT_TYPES list:

    • Privilege grants to share objects are lost.

To avoid dropping users or roles in the target account:

1. In the source account, manually recreate any users or roles that exist only in the target account before the initial replication.

2. In the target account, link matching objects with the same name in both accounts using the SYSTEM$LINK_ACCOUNT_OBJECTS_BY_NAME function.

Amazon S3¶

The configuration process depends on how you set up event notifications. For example, suppose you have an auto-ingest pipe that relies on an Amazon Simple Notification Service (SNS) topic to publish messages about the Snowflake stage location.

When you replicate the pipe to a target account, Snowflake automatically creates a new Amazon Simple Queue Service (SQS) queue. You must subscribe this SQS queue for your target account to the SNS topic to get notifications about the stage location.

• If you use Amazon S3 Event Notifications with Amazon Simple Queue Service (SQS), follow the instructions in Step 4: Configure Event Notifications.

  Important

  To ensure that the pipe has not missed any notifications, you should refresh the pipe after switching to the new SQS queue.

• If you use Amazon Simple Notification Service (SNS), see Subscribing the Snowflake SQS Queue to your SNS topic.

• If you use Amazon EventBridge, see Option 3: Setting up Amazon EventBridge to Automate Snowpipe.

Microsoft Azure Blob Storage¶

A pipe that automatically loads data from files located on a stage in Microsoft Azure blob storage requires an Event Grid subscription, storage queue, and a notification integration bound to the storage queue. A secondary pipe in a target account needs a separate Event Grid, storage queue, and notification integration bound to the storage queue. The Event Grid in both source and target accounts must be configured as endpoints for the same Azure Storage source.

See the diagram below for configuration details:

Follow the detailed instructions in Configuring Automation With Azure Event Grid to configure an Event Grid subscription and notification integration.

External Stage for Google Cloud Storage¶

A pipe that automatically loads data from files located in Google Cloud Storage requires a Google Pub/Sub subscription and a notification integration that references that subscription. Each replicated pipe in a target account also requires a Google Pub/Sub subscription and a notification integration that references that subscription. The Pub/Sub subscription in each source and target account must be subscribed to the same Pub/Sub Topic that receives notifications from the Google Cloud Storage source.

See the diagram below for configuration details:

For detailed instructions on creating a Pub/Sub subscription and notification integration, see Configuring Automation Using GCS Pub/Sub.

Monitor the Progress of a Replication Group or Failover Group Refresh¶

To monitor the progress of a replication or failover group refresh, query the REPLICATION_GROUP_REFRESH_PROGRESS, REPLICATION_GROUP_REFRESH_PROGRESS_BY_JOB table function (in the Snowflake Information Schema).

SELECT PHASE_NAME, START_TIME, END_TIME, PROGRESS, DETAILS
  FROM TABLE(information_schema.replication_group_refresh_progress('myfg'));

Replication History¶

To view the replication history of a specific replication or failover group within a specified date range, query one of the following:

• REPLICATION_GROUP_REFRESH_HISTORY table function (in the Snowflake Information Schema).

• REPLICATION_GROUP_REFRESH_HISTORY View (in Account Usage).

SELECT PHASE_NAME, START_TIME, END_TIME, TOTAL_BYTES, OBJECT_COUNT
  FROM TABLE(information_schema.replication_group_refresh_history('myfg'))
  WHERE START_TIME >= current_date - interval '7 days';

SELECT REPLICATION_GROUP_NAME, PHASE_NAME, START_TIME, END_TIME, TOTAL_BYTES, OBJECT_COUNT
  FROM snowflake.account_usage.replication_group_refresh_history
  WHERE START_TIME >= date_trunc('month', current_date());

Monitor Replication Costs¶

To monitor credit usage for replication, query one of the following:

• REPLICATION_GROUP_USAGE_HISTORY View (in Account Usage).

• REPLICATION_GROUP_USAGE_HISTORY table function (in the Snowflake Information Schema).

SELECT start_time, end_time, replication_group_name, credits_used, bytes_transferred
  FROM table(information_schema.replication_group_usage_history(date_range_start=>dateadd('day', -7, current_date())));

SELECT start_time, 
  end_time, 
  replication_group_name, 
  credits_used, 
  bytes_transferred
FROM snowflake.account_usage.replication_group_usage_history
WHERE start_time >= DATE_TRUNC('month', CURRENT_DATE());

Monitor Replication Costs for Databases¶

The cost for replication for an individual database included in a replication or failover group can be calculated by retrieving the number of copied bytes for the database and associating it with the credits used.

Example¶

In the examples below, the database mydb is included in the failover group myfg. The database mydb contains the table mytable.