Apache Iceberg™ tables¶

Getting started¶

To get started with Iceberg tables, see Tutorial: Create your first Apache Iceberg™ table.

How it works¶

This section provides information specific to working with Iceberg tables in Snowflake. To learn more about the Iceberg table format specification, see the official Apache Iceberg documentation and the Iceberg Table Spec.

• Data storage

• Catalog

• Metadata and snapshots

• Cross-cloud/cross-region support

• Billing

Catalog options¶

Snowflake supports the following Iceberg catalog options:

• Use Snowflake as the Iceberg catalog

• Use an external Iceberg catalog

The following table summarizes the differences between these catalog options.

Use Snowflake as the catalog¶

An Iceberg table that uses Snowflake as the Iceberg catalog (Snowflake-managed Iceberg table) provides full Snowflake platform support with read and write access. The table data and metadata are stored in external cloud storage, which Snowflake accesses using an external volume. Snowflake handles all life-cycle maintenance, such as compaction, for the table.

Use an external catalog¶

An Iceberg table that uses an external catalog provides limited Snowflake platform support with read-only access. With this table type, Snowflake uses a catalog integration to retrieve information about your Iceberg metadata and schema.

You can use this option to create an Iceberg table for the following sources:

• AWS Glue Data Catalog

• Iceberg metadata files in object storage

• Delta table files in object storage

• Open Catalog

• Remote Iceberg REST catalog

Snowflake does not assume any life-cycle management on the table.

The table data and metadata are stored in external cloud storage, which Snowflake accesses using an external volume.

The following diagram shows how an Iceberg table uses a catalog integration with an external Iceberg catalog.

Considerations and limitations¶

The following considerations and limitations apply to Iceberg tables, and are subject to change:

Clouds and regions

• Iceberg tables are available for all Snowflake accounts, on all cloud platforms and in all regions.

• Cross-cloud/cross-region tables are supported when you use an external catalog. For more information, see Cross-cloud/cross-region support.

Iceberg

• Versions 1 and 2 of the Apache Iceberg specification are supported, excluding the following features:

  • Row-level deletes (either position deletes or equality deletes). However, tables that use Snowflake as the catalog support Snowflake DELETE statements.

  • Using the history.expire.min-snapshots-to-keep table property to specify the default minimum number of snapshots to keep. For more information, see Metadata and snapshots.

• Iceberg partitioning with the bucket transform function impacts performance for queries that use conditional clauses to filter results.

• For Iceberg tables that aren’t managed by Snowflake, be aware of the following:

  • Time travel to any snapshot generated after table creation is supported as long as you periodically refresh the table before the snapshot expires.

  • Converting a table that has an un-materialized identity partition column isn’t supported. An un-materialized identity partition column is created when a table defines an identity transform using a source column that doesn’t exist in a Parquet file.

  • For row-level deletes (preview feature):

    • Snowflake supports position deletes only.

    • For the best read performance when you use row-level deletes, perform regular compaction and table maintenance to remove old delete files. For information, see Maintain tables that use an external catalog.

    • The following features aren’t currently supported when you use row-level deletes:

      • Automated refresh

      • Creation of materialized views, search optimization, or streams on dynamic tables, dynamic Iceberg tables, or Iceberg tables with existing position deletes.

      • Dynamic tables

      • Refresh of Iceberg snapshot with position deletes on dynamic tables, dynamic Iceberg tables, or Iceberg tables with either active or deleted materialized views, search optimization, or streams.

      • Search optimization

      • Streams

File formats

• Iceberg tables support Apache Parquet files.

• Parquet files that use the unsigned integer logical type aren’t supported.

• For Parquet files that use the LIST logical type, be aware of the following:

  • The three-level annotation structure with the element keyword is supported. For more information, see Parquet Logical Type Definitions. If your Parquet file uses an obsolete format with the array keyword, you must regenerate your data based on the supported format.

External volumes

• You can’t access the cloud storage locations in external volumes using a storage integration.

• You must configure a separate trust relationship for each external volume that you create.

• You can use outbound private connectivity to access Snowflake-managed Iceberg tables and Iceberg tables that use a catalog integration for object storage, but cannot use it to access Iceberg tables that use other catalog integrations.

• After you create a Snowflake-managed table, the path to its files in external storage does not change, even if you rename the table.

• Snowflake can’t support external volumes with S3 bucket names that contain dots (for example, my.s3.bucket). S3 doesn’t support SSL for virtual-hosted-style buckets with dots in the name, and Snowflake uses virtual-host-style paths and HTTPS to access data in S3.

Metadata files

• The metadata files don’t identify the most recent snapshot of an Iceberg table.

• You can’t modify the location of the data files or snapshot using the ALTER ICEBERG TABLE command. To modify either of these settings, you must recreate the table (using the CREATE OR REPLACE ICEBERG TABLE syntax).

• For tables that use an external catalog:

  • Ensure that manifest files don’t contain duplicates. If duplicate files are present in the same snapshot, Snowflake returns an error that includes the path of the duplicate file.

  • You can’t create a table if the Parquet metadata contains invalid UTF-8 characters. Ensure that your Parquet metadata is UTF-8 compliant.

• Snowflake detects corruptions and inconsistencies in Parquet metadata produced outside of Snowflake, and surfaces issues through error messages.

  It’s possible to create, refresh, or query externally managed (or converted) tables, even if the table metadata is inconsistent. When writing Iceberg data, ensure that the table’s metadata statistics (for example, RowCount or NullCount) match the data content.

• For tables that use Snowflake as the catalog, Snowflake processes DDL statements individually and produces metadata in a way that might differ from other catalogs. For more information, see DDL statements.

Clustering

Clustering support depends on the type of Iceberg table.

Table type	Notes
Tables that use Snowflake as the Iceberg catalog	Set a clustering key by using either the CREATE ICEBERG TABLE or the ALTER ICEBERG TABLE command. To set or manage a clustering key, see CREATE ICEBERG TABLE (Snowflake as the Iceberg catalog) and ALTER ICEBERG TABLE.
Tables that use an external catalog	Clustering is not supported.
Converted tables	Snowflake only clusters files if they were created after converting the table, or if the files have since been modified using a DML statement.

Delta

• Snowflake supports Delta reader version 2 and can read all tables written by engines using Delta Lake version 2.2.0.

• Snowflake streams aren’t supported for Iceberg tables created from Delta table files with partition columns. However, insert-only streams for tables created from Delta files without partition columns are supported.

• Dynamic tables aren’t supported on Iceberg tables created from Delta table files.

• Snowflake doesn’t support creating Iceberg tables from Delta table definitions in the AWS Glue Data Catalog.

• Parquet files (data files for Delta tables) that use any of the following features or data types aren’t supported:

  • Field IDs.

  • The INTERVAL data type.

  • The DECIMAL data type with precision higher than 38.

  • LIST or MAP types with one-level or two-level representation.

  • Unsigned integer types (INT(signed = false)).

  • The FLOAT16 data type.

• You can use the Parquet physical type int96 for TIMESTAMP, but Snowflake doesn’t support int96 for TIMESTAMP_NTZ.

• For more information about Delta data types and Iceberg tables, see Delta data types.

• Snowflake processes a maximum of 1000 Delta commit files each time you refresh a table using CREATE/ALTER … REFRESH. If your table has over 1000 commit files, you can do additional manual refreshes. Each time, the refresh process continues from where the last one stopped.

  Note

  Snowflake uses Delta checkpoint files when creating an Iceberg table. The 1,000 commit file limit only applies to commits after the latest checkpoint.

  When you refresh an existing table, Snowflake processes Delta commit files, but not checkpoint files. If table maintenance removes stale log and data files for the source Delta table, you should refresh Delta-based Iceberg tables in Snowflake more frequently than the retention period of Delta logs and data files.

• Generating Iceberg metadata using the SYSTEM$GET_ICEBERG_TABLE_INFORMATION function isn’t supported.

• The following Delta Lake features aren’t currently supported: Row tracking, deletion vector files, change data files, change metadata, DataChange, CDC, protocol evolution.

Automated refresh

• When automated refresh is enabled, you can’t manually refresh the table metadata. To perform a manual refresh, turn off automated refresh first.

• For catalog integrations created before Snowflake version 8.22 (or 9.2 for Delta-based tables), you must manually set the REFRESH_INTERVAL_SECONDS parameter before you enable automated refresh on tables that depend on that catalog integration. For instructions, see ALTER CATALOG INTEGRATION … SET AUTO_REFRESH.

• If your table is empty, insert data into the table and then perform a manual refresh in Snowflake before you enable automated refresh to avoid undefined behavior.

• For catalog integrations for object storage, automated refresh is only supported for integrations with TABLE_FORMAT = DELTA.

• For tables with frequent updates, using a shorter polling interval (REFRESH_INTERVAL_SECONDS) can cause performance degradation.

Access by third-party clients to Iceberg data, metadata

• Third-party clients can’t append to, delete from, or upsert data to Iceberg tables that use Snowflake as the catalog.

S3-compatible storage

Iceberg tables that use S3-compatible storage are cross-region tables and don’t support the following actions:

• Using Snowflake as the Iceberg catalog or converting a table to use Snowflake as the Iceberg catalog.

• Retrieving information about the latest refreshed snapshot with the SYSTEM$GET_ICEBERG_TABLE_INFORMATION function.

Unsupported features

The following Snowflake features aren’t currently supported for all Iceberg tables:

• Collation

• Fail-safe

• Hybrid tables

• Listings that enable Cross-Cloud Auto-Fulfillment.

• Query Acceleration Service

• Replication of Iceberg tables, external volumes, or catalog integrations

• Search optimization service

• Snowflake encryption

• Snowflake Native App Framework

• Snowflake schema evolution

• Tagging using the ASSOCIATE_SEMANTIC_CATEGORY_TAGS stored procedure

• Temporary and transient tables

The following features aren’t supported for Iceberg tables that use an external catalog:

• Cloning

• Clustering

• Standard and append-only streams. Insert-only streams are supported.