Automating Continuous Data Loading Using Cloud Messaging¶
Automated data loads leverage event notifications for cloud storage to inform Snowpipe of the arrival of new data files to load. Snowpipe copies the files into a queue, from which they are loaded into the target table in a continuous, serverless fashion based on parameters defined in a specified pipe object.
Currently, this automated Snowpipe workflow supports data files staged in Amazon S3 or Microsoft Azure blob storage. Support for Google Cloud Storage is planned.
Automated Snowpipe uses event notifications to determine when new files arrive in monitored cloud storage and are ready to load. Notifications identify the cloud storage event and include a list of the file names. They do not include the actual data in the files.