Enabling Snowpipe error notifications for Google Pub/Sub¶

Step 1: Creating the Pub/Sub Topic¶

Create a Pub/Sub topic that can receive error notification messages from Snowflake, or reuse an existing topic. You can create the topic using Cloud Shell or Cloud SDK. For more information, see Create and use topics in the Pub/Sub documentation.

For example, execute the following command to create an empty topic:

$ gsutil notification create -t <topic>

If the topic already exists, the command uses it; otherwise, a new topic is created.

Step 2: Creating the Pub/Sub Subscription¶

Optionally create a subscription to the Pub/Sub topic to retrieve error notifications. You can create a subscription with pull delivery using the Cloud Console, gcloud command-line tool, or the Cloud Pub/Sub API. For instructions, see Managing topics and subscriptions in the Pub/Sub documentation.

Step 3: Creating a Notification Integration in Snowflake¶

Create a notification integration using the CREATE NOTIFICATION INTEGRATION command. The notification integration references your Pub/Sub topic. Snowflake associates the notification integration with a Goodle Cloud Platform (GCP) service account created for your account. Snowflake creates a single service account that is referenced by all GCP notification integrations in your Snowflake account.

CREATE NOTIFICATION INTEGRATION <integration_name>
  ENABLED = TRUE
  TYPE = QUEUE
  DIRECTION = OUTBOUND
  NOTIFICATION_PROVIDER = GCP_PUBSUB
  GCP_PUBSUB_TOPIC_NAME = '<topic_id>'

Where:

• integration_name is the name of the new integration.

• topic_id is the Pub/Sub topic to which Snowflake sends error notifications. For more information, see Step 1: Creating the Pub/Sub Topic (in this topic).

For example:

CREATE NOTIFICATION INTEGRATION my_notification_int
  TYPE = QUEUE
  DIRECTION = OUTBOUND
  NOTIFICATION_PROVIDER = GCP_PUBSUB
  ENABLED = true
  GCP_PUBSUB_TOPIC_NAME = 'projects/sdm-prod/topics/mytopic';

Step 4: Granting Snowflake Access to the Pub/Sub Subscription¶

1. Execute the DESCRIBE INTEGRATION command to retrieve the Snowflake service account ID:

   DESC NOTIFICATION INTEGRATION <integration_name>;
   

   Copy

   Where:

   • integration_name is the name of the integration you created in “Step 1: Create a Notification Integration in Snowflake”.

   For example:

   DESC NOTIFICATION INTEGRATION my_notification_int;
   

   Copy

2. Record the service account name in the GCP_PUBSUB_SERVICE_ACCOUNT column, which has the following format:

   <service_account>@<project_id>.iam.gserviceaccount.com
   

   Copy

3. Log into the Google Cloud Platform Console as a project editor.

4. From the home dashboard, choose Big Data » Pub/Sub » Subscriptions.

5. Select the subscription to configure for access.

6. Click SHOW INFO PANEL in the upper-right corner. The information panel for the subscription slides out.

7. In the Add members field, search for the service account name you recorded.

8. From the Select a role dropdown, select Pub/Sub Publisher.

9. Click the Add button. The service account name is added to the Pub/Sub Publisher role dropdown in the information panel.

Step 5: Enabling error notifications in pipes¶

A single notification integration can be shared by multiple pipes. The body of error messages identifies the pipe, external stage and path, and file where the error originated, among other details.

To enable error notifications for a pipe, specify an ERROR_INTEGRATION parameter value.

Note that operating on any object in a schema also requires the USAGE privilege on the parent database and schema.

For instructions on creating a custom role with a specified set of privileges, see Creating custom roles.

For general information about roles and privilege grants for performing SQL actions on securable objects, see Overview of Access Control.

CREATE PIPE <name>
  AUTO_INGEST = TRUE
  [ INTEGRATION = '<string>' ]
  ERROR_INTEGRATION = <integration_name>
  AS <copy_statement>

CREATE PIPE mypipe
  AUTO_INGEST = TRUE
  INTEGRATION = 'my_storage_int'
  ERROR_INTEGRATION = my_notification_int
  AS
  COPY INTO mydb.public.mytable
  FROM @mydb.public.mystage;

ALTER PIPE <name> SET ERROR_INTEGRATION = <integration_name>;

ALTER PIPE mypipe SET ERROR_INTEGRATION = my_notification_int;