Iceberg tables¶
Preview Feature — Open
Available to all accounts.
An Iceberg table uses the Apache Iceberg open table format specification, which provides an abstraction layer on data files stored in open formats and supports features such as:
ACID (atomicity, consistency, isolation, durability) transactions
Schema evolution
Hidden partitioning
Table snapshots
Iceberg tables for Snowflake combine the performance and query semantics of regular Snowflake tables with external cloud storage that you manage. They are ideal for existing data lakes that you cannot, or choose not to, store in Snowflake.
Snowflake supports Iceberg tables that use the Apache Parquet file format.
How Iceberg tables work¶
This section provides information specific to working with Iceberg tables in Snowflake. To learn more about the Iceberg table format specification, see the official Apache Iceberg documentation and the Iceberg Table Spec.
Data storage¶
Iceberg tables store their data and metadata files in an external cloud storage location (Amazon S3, Google Cloud Storage, or Azure Storage). The external storage is not part of Snowflake. You are responsible for all management of the external cloud storage location. Snowflake connects to your storage location using an external volume.
Iceberg tables incur no Snowflake storage costs. For more information, see Billing.
External volumes¶
An external volume is a named, account-level Snowflake object that stores an identity and access management (IAM) entity for your external cloud storage. Snowflake securely connects to your cloud storage with an external volume to access table data, Iceberg metadata, and manifest files that store the table schema, partitions, and other metadata.
A single external volume can support one or more Iceberg tables.
To set up an external volume for Iceberg tables, see Configure an external volume for Iceberg tables.
Iceberg catalog¶
An Iceberg catalog enables a compute engine to manage and load Iceberg tables. The catalog forms the first architectural layer in the Iceberg table specification and must support:
Storing the current metadata pointer for one or more Iceberg tables. A metadata pointer maps a table name to the location of that table’s current metadata file.
Performing atomic operations so that you can update the current metadata pointer for a table.
To learn more about Iceberg catalogs, see the Apache Iceberg documentation.
Snowflake supports different catalog options. For example, you can use Snowflake as the Iceberg catalog, or use a catalog integration to connect Snowflake to an external Iceberg catalog like AWS Glue or to Iceberg metadata files in object storage.
Catalog integrations¶
A catalog integration is a named, account-level Snowflake object that defines the source of metadata and schema for an Iceberg table when you do not use Snowflake as the Iceberg catalog.
A single catalog integration can support one or more Iceberg tables.
To set up a catalog integration for Iceberg tables, see Configure a catalog integration for Iceberg tables.
Metadata and snapshots¶
For tables that use Snowflake as the catalog, Snowflake automatically generates the metadata on a periodic basis. It then writes the metadata to the table’s Parquet files on your external volume.
Iceberg uses a snapshot-based querying model, where data files are mapped using manifest and metadata files. A snapshot represents the state of a table at a point in time and is used to access the complete set of data files in the table.
Snowflake expires older Iceberg table snapshots to reduce the size of table metadata.
To determine the maximum age of snapshots, Snowflake retrieves the value of history.expire.max-snapshot-age-ms
from the current metadata file, converts the value to days (rounding down), and stores it in the
DATA_RETENTION_TIME_IN_DAYS parameter.
If Snowflake does not find history.expire.max-snapshot-age-ms in the metadata file, or cannot parse the value, it sets
DATA_RETENTION_TIME_IN_DAYS to a default value of 5 days (the default Apache Iceberg value).
Note
Using the history.expire.min-snapshots-to-keep
table property to specify
the default minimum number of snapshots is not supported.
Billing¶
Snowflake bills your account for virtual warehouse (compute) usage and cloud services when you work with Iceberg tables.
Snowflake does not bill your account for the following:
Iceberg table storage costs. Your cloud storage provider bills you directly for data storage usage.
Active bytes used by Iceberg tables. However, the TABLE_STORAGE_METRICS View displays ACTIVE_BYTES for Iceberg tables to help you track how much storage a table occupies.
Iceberg catalog options¶
When you create an Iceberg table in Snowflake, you can use Snowflake as the Iceberg catalog or you can use a catalog integration.
The following table summarizes the differences between these catalog options.
Read access |
✔ |
✔ |
Write access |
✔ |
❌ For full platform support, you can convert the table to use Snowflake as the catalog. |
Data and metadata storage |
External volume (cloud storage) |
External volume (cloud storage) |
Full platform support |
✔ |
❌ |
Works with the Snowflake Iceberg Catalog SDK |
✔ |
✔ |
Use Snowflake as the Iceberg catalog¶
An Iceberg table that uses Snowflake as the Iceberg catalog provides full Snowflake platform support with read and write access. The table data and metadata are stored in external cloud storage, which Snowflake accesses using an external volume. Snowflake handles all life-cycle maintenance, such as compaction, for the table.
Use a catalog integration¶
An Iceberg table that uses a catalog integration provides limited Snowflake platform support with read-only access. The table data and metadata are stored in external cloud storage, which Snowflake accesses using an external volume. With this table type, Snowflake uses the catalog integration to retrieve information about your Iceberg metadata and schema. Snowflake does not assume any life-cycle management on the table.
You can use this option to create an Iceberg table that uses an external Iceberg catalog, such as AWS Glue, or to create a table from Iceberg metadata files in object storage. The following diagram shows how an Iceberg table uses a catalog integration with an external Iceberg catalog.
Considerations and limitations¶
The following considerations and limitations apply to Iceberg tables, and are subject to change:
Iceberg
Versions 1 and 2 of the Apache Iceberg specification are supported, excluding the following features:
Row-level deletes (either position deletes or equality deletes).
Using the
history.expire.min-snapshots-to-keeptable property to specify the default minimum number of snapshots to keep. For more information, see Snapshot Expiration.Iceberg partitioning with the
buckettransform function impacts performance for queries that use conditional clauses to filter results.For Iceberg tables that are not managed by Snowflake, time travel to any snapshot generated after table creation is supported as long as you periodically refresh the table before the snapshot expires.
File formats
Support is limited to Apache Parquet files.
Parquet files that use the unsigned integer logical type are not supported.
External volumes
Cross-cloud and cross-region Iceberg tables are not currently supported. Your external volume must specify a storage location with the same cloud provider (in the same region) that hosts your Snowflake account.
You must access the cloud storage locations in external volumes using direct credentials. Storage integrations are not supported.
The trust relationship must be configured separately for each external volume that you create.
Metadata files
The metadata files do not identify the most recent snapshot of an Iceberg table.
You cannot modify the location of the data files or snapshot using the ALTER ICEBERG TABLE command. To modify either of these settings, you must recreate the table (using the CREATE OR REPLACE ICEBERG TABLE syntax).
Snowflake features
The following features and actions are currently not supported on Iceberg tables:
Creating a clone from an Iceberg table. In addition, clones of databases and schemas do not include Iceberg tables.
Automatically applying tags using the ASSOCIATE_SEMANTIC_CATEGORY_TAGS stored procedure.
Snowflake schema evolution. However, Iceberg tables do support Iceberg schema evolution.
Creating temporary or transient Iceberg tables.
Replicating Iceberg tables, external volumes, or catalog integrations.
Granting or revoking privileges for Iceberg tables, external volumes, or catalog integrations to or from a share.
Querying historical data is supported for Iceberg tables.
Clustering support depends on the type of Iceberg table.
Table type
Notes
Tables that use Snowflake as the Iceberg catalog
Set a clustering key by using either the CREATE ICEBERG TABLE or the ALTER ICEBERG TABLE command. To set or manage a clustering key, see CREATE ICEBERG TABLE and ALTER ICEBERG TABLE.
Tables that use a catalog integration
Clustering is not supported.
Converted tables
Snowflake only clusters files if they were created after converting the table, or if the files have since been modified using a DML statement.
Access by third-party clients to Iceberg data, metadata
Third-party clients cannot read Iceberg tables without first exporting the metadata.
Third-party clients cannot append to, delete from, or upsert data to Iceberg tables.
Next Topics: