Metadata and retention for Apache Iceberg™ tables¶

Creation¶

Snowflake generates metadata for version 2 of the Apache Iceberg specification on a periodic basis, and writes the metadata to files on your external volume. Each new metadata file contains all DML or DDL changes since the last Snowflake-generated metadata file was created.

You can also create metadata on demand by using the SYSTEM$GET_ICEBERG_TABLE_INFORMATION function. For instructions, see Generate snapshots of DML changes.

For information about locating metadata files, see Data and metadata directories.

Viewing metadata creation history¶

To access a full history of metadata generation attempts, view the query history for your account and filter the results. Search for the SYSTEM$GET_ICEBERG_TABLE_INFORMATION function name in the SQL text.

Snowflake internally uses the same SYSTEM$GET_ICEBERG_TABLE_INFORMATION function to generate table metadata. Attempts made by Snowflake appear under the user called SYSTEM in the query history. The STATUS column in the query history indicates whether metadata was successfully generated.

For viewing options, see Monitor query activity with Query History.

Deletion¶

Snowflake deletes Iceberg metadata from your external cloud storage when the following events occur:

• After you drop a table.

• When the Iceberg metadata refers to snapshots or table data that has expired.

Deletion doesn’t occur immediately after the data retention period expires. As a result, metadata storage might incur costs with your cloud storage provider for longer than a table’s lifetime.

Delta-based tables¶

For Iceberg tables created from Delta table files, Snowflake automatically writes Iceberg metadata to your external storage if you configure your external volume with write access (see ALLOW_WRITES). For more information about the write location, see Data and metadata directories.

To prevent Snowflake from writing Iceberg metadata, you can set the ALLOW_WRITES parameter to FALSE on your external volume as long as no Snowflake-managed Iceberg tables use the same external volume.

“Hidden” partitioning¶

“Hidden” partitioning for Apache Iceberg™ is metadata-based and adaptable. Iceberg produces partition values based on transforms that you define when you create a table. When they read from a partitioned table, Iceberg engines use the partition values defined in your table metadata to efficiently identify relevant data.

This option is the default. With this option, Snowflake stores your Parquet data files by using a flat directory layout.

To create a partitioned Iceberg table that uses hidden partitioning, include the PARTITION BY clause with one or more partition transforms in your regular CREATE ICEBERG TABLE statement.

For an example, see Create an Iceberg table in a catalog-linked database.

Partitioning with hierarchical paths¶

With this option, Snowflake writes data to partitioned Iceberg tables by using a hierarchical path layout for Parquet data files. Partitioning information is included in the file paths and the values are based on transforms that you define when you create a table. This layout is also called “Hive-style” partitioning. You might use this option for interoperability between Snowflake and external engines that support partitioned writes with hierarchical paths.

Here’s an example of a data file stored under a hierarchical path:

s3://my-bucket/iceberg/db_sales/orders/data/country=US/year=2025/month=02/day=21/part-00023.parquet

For more information on the layout of the data and metadata directories for tables that use hierarchical paths, see File management.

Partitioning support matrix¶

The following table shows which features and actions are supported for each type of partitioned Iceberg table, and indicates compliance with version 2 of the Apache Iceberg specification. The table shows support for both hidden partitioning and partitioning with hierarchical paths.

Partitioning considerations¶

Consider the following before you use partitioned writes for Iceberg tables:

• If you use an external engine to add, drop, or replace a partition field in an externally managed table, Snowflake writes data according to the latest partition specification.

• The GET_DDL function doesn’t include the PARTITION BY clause in its output.

• The sum of the sizes of the outputs for all partition transforms can’t exceed 1024 bytes for a single row.

• Because partition evolution isn’t supported for Snowflake-managed tables, you must drop the table and create a new one with partitioning.

• The DAY(), MONTH(), YEAR() partition transform parameters, which you specify within the PARTITION BY clause under table properties, are part of the Iceberg specification. For multiple days, months, or years, the partition expression parameter returns a partition for each calendar day, month, or year. For example, when the DAY() transform is used on a timestamp column that has 2 months of data, 61 partitions are created.

  In contrast, the DAY(), MONTH(), YEAR() functions in Snowflake are part of the SQL standard. For multiple days, months, or years, these functions extract the corresponding day, month, or year part from a date or timestamp. For example, when the DAY() function is used on a timestamp column that has multiple months of data, this function returns a day of the month ranging from 1 to 31.

• You can’t use the ALTER ICEBERG TABLE command to modify the PATH_LAYOUT property for an existing table.

• For partitioning with hierarchical paths:

  • For float values, Snowflake and external engines might behave differently.

  • Snowflake can’t guarantee that the paths Snowflake writes will match the paths that external query engines write.

    Snowflake can’t guarantee this because when a query engine writes a hierarchical path, the query engine must serialize values into a string and insert the resulting value into the path. The Apache Iceberg table specification doesn’t define a standard serialization method, so different engines might implement different methods.

    For example, Snowflake doesn’t encode the ~ character but Apache Spark™ encodes this character as %7E.

  • Snowflake always writes the hierarchical paths directly under the /data directory in your external cloud storage.