Automated Directory Table Metadata Refreshes

The metadata for an directory table can be refreshed automatically using the following event notification service for each cloud storage service:

The refresh operation synchronizes the metadata with the latest set of associated files in the external stage and path, i.e.:

  • New files in the path are added to the table metadata.

  • Changes to files in the path are updated in the table metadata.

  • Files no longer in the path are removed from the table metadata.

Note

Currently, the ability to automatically refresh the metadata is not available for directory tables for either internal stages or external stages that reference Google Cloud Storage buckets. For these types of stages, you must manually refresh the directory table metadata. For instructions, see Manually Refreshing Directory Table Metadata.

We suggest following our best practices for staging your data files and periodically executing an ALTER STAGE … REFRESH statement to register any missed files. For satisfactory performance, we also recommend using a selective path prefix with ALTER STAGE to reduce the number of files that need to be listed and checked if they have been registered already (e.g. bucket_name/YYYY/MM/DD/ or even bucket_name/YYYY/MM/DD/HH/ depending on your volume).

For setup instructions, see the topic for the cloud storage service where your files are located.

The following table indicates which cloud storage services are supported for automatically refreshing directory table metadata to your Snowflake account, based on the cloud platform that hosts your account:

Snowflake Account Host

Amazon S3

Google Cloud Storage

Microsoft Azure Blob storage

Microsoft Data Lake Storage Gen2

Microsoft Azure General-purpose v2

Amazon Web Services

Google Cloud Platform

Microsoft Azure

Next Topics: