Access externally managed Apache Iceberg™ tables with an external engine through Snowflake Horizon Catalog¶

This topic describes how to access externally managed Apache Iceberg™ tables through Snowflake Horizon Catalog by using external query engines. The tables can be in a catalog-linked database or in a standard Snowflake database. To create an externally managed table in a standard Snowflake database, see Create an Iceberg table in a standard Snowflake database.

With this feature, you can use the Horizon Iceberg REST Catalog (IRC) API to access and operate on Iceberg tables from external engines such as Apache Spark™, Trino, DuckDB, or PyIceberg. This extends the existing Horizon IRC support for Snowflake-managed Iceberg tables so that one catalog endpoint covers Snowflake-managed tables and externally managed tables.

Overview¶

Snowflake Horizon Catalog provides governance for Snowflake-managed and externally managed Iceberg tables. By pointing your external engines to the Horizon IRC endpoint, you can:

• Use a single catalog endpoint for Snowflake-managed and externally managed Iceberg tables.
• Enforce Snowflake RBAC before a request is sent to the remote catalog.
• Access and operate on externally managed tables without copying governance configuration into each engine.

Where the table lives changes what a catalog operation does:

• In a catalog-linked database, reads, DML, and DDL are sent to the remote catalog, and Snowflake keeps a matching schema or table.
• In a standard Snowflake database, reads and DML on an externally managed table are sent to the remote catalog. Other catalog operations change the Snowflake object only.

Prerequisites¶

Before you access externally managed tables through Horizon IRC, complete the following:

• Create the tables in one of these places:
  • A catalog-linked database. For instructions, see Use a catalog-linked database for Apache Iceberg™ tables.
  • A standard Snowflake database. For instructions, see Create an Iceberg table in a standard Snowflake database.
• Configure the external engine for Horizon Catalog. For instructions, see Access Apache Iceberg™ tables with an external engine through Snowflake Horizon Catalog.
• To use credentials vended by Horizon Catalog, the database needs an external volume, and the catalog integration must use ACCESS_DELEGATION_MODE = EXTERNAL_VOLUME_CREDENTIALS. Credential vending for externally managed tables is disabled by default. For more information, see Credential vending.

Supported operations¶

The operations below depend on whether the table is in a catalog-linked database or a standard Snowflake database.

Catalog-linked database¶

Each operation also depends on support in your remote catalog. If your remote catalog doesn’t implement the corresponding Iceberg REST operation, that operation isn’t available through Horizon IRC.

Standard Snowflake database¶

In a standard database, Horizon IRC behavior depends on the table. loadTable and updateTable for an externally managed table are sent to the remote catalog. Every other operation follows Snowflake semantics and doesn’t change the remote catalog.

External engines use the Snowflake database, schema, and table names. Horizon IRC doesn’t expose the table’s name in the remote catalog.

Connect an external engine to externally managed Iceberg tables¶

To access these tables through Horizon IRC, use the same connection workflow described in Access Apache Iceberg™ tables with an external engine through Snowflake Horizon Catalog. When you configure your external engine, set the warehouse property to the Snowflake database name. That name is the catalog-linked database or the standard database that contains the tables.

The name that you pass must match the Snowflake identifier exactly: the form returned by SHOW DATABASES. If the database was created without quotes (the common case), use the upper-cased form. If it was created with a quoted identifier, use the exact case it was created with. A mismatch returns a 404 error.

Example: Spark configuration¶

The following example shows how to configure Apache Spark™ to connect through Horizon IRC:

CATALOG_URI = "https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog"
HORIZON_SESSION_ROLE = "session:role:<role>"
CATALOG_NAME = "<database_name>"  # Snowflake database name, as returned by SHOW DATABASES
ICEBERG_VERSION = "1.10.1"

spark = (
    SparkSession.builder
    .appName("HorizonExternalIceberg")
    .master("local[*]")
    .config(
        "spark.jars.packages",
        f"org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{ICEBERG_VERSION},"
        f"org.apache.iceberg:iceberg-aws-bundle:{ICEBERG_VERSION}"
    )
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    .config("spark.sql.defaultCatalog", CATALOG_NAME)
    .config(f"spark.sql.catalog.{CATALOG_NAME}", "org.apache.iceberg.spark.SparkCatalog")
    .config(f"spark.sql.catalog.{CATALOG_NAME}.type", "rest")
    .config(f"spark.sql.catalog.{CATALOG_NAME}.uri", CATALOG_URI)
    .config(f"spark.sql.catalog.{CATALOG_NAME}.warehouse", CATALOG_NAME)
    .config(f"spark.sql.catalog.{CATALOG_NAME}.credential", "<your_PAT_token>")
    .config(f"spark.sql.catalog.{CATALOG_NAME}.scope", HORIZON_SESSION_ROLE)
    .config(f"spark.sql.catalog.{CATALOG_NAME}.header.X-Iceberg-Access-Delegation", "vended-credentials")
    .config("spark.sql.iceberg.vectorization.enabled", "false")
    .getOrCreate()
)

Where:

• <account_identifier> is the account identifier for your Snowflake account, in the format <organization_name>-<account_name>.
• <role> is the Snowflake role that has the appropriate privileges on the database and its tables.
• <database_name> is the name of your catalog-linked database or standard Snowflake database. It must match the Snowflake identifier exactly: the form returned by SHOW DATABASES. Unquoted identifiers resolve to upper case; quoted identifiers preserve the exact case that was used at creation.
• <your_PAT_token> is the programmatic access token (PAT) for authentication. For more authentication options, see Access Apache Iceberg™ tables with an external engine through Snowflake Horizon Catalog.

Access tables¶

After configuring your external engine, you can access tables in the database. The examples use SALES.ORDERS, an externally managed Iceberg table.

List namespaces¶

spark.sql("SHOW NAMESPACES").show()

List tables in a namespace¶

spark.sql("USE NAMESPACE SALES")
spark.sql("SHOW TABLES").show()

Query a table¶

spark.sql("SELECT order_id, status FROM SALES.ORDERS LIMIT 10").show()

Write to a table (DML)¶

spark.sql("INSERT INTO SALES.ORDERS VALUES (1001, 'shipped')")

spark.sql("UPDATE SALES.ORDERS SET status = 'delivered' WHERE order_id = 1001")

spark.sql("DELETE FROM SALES.ORDERS WHERE order_id = 1001")

Configure access control¶

The role specified in the session scope must have the appropriate privileges:

• Read access: The role must have USAGE on the database, USAGE on the schema, and the privileges described in Configure read access to your Iceberg tables on the table.
• Write access (DML): The role must have the privileges described in Configure write access to your Iceberg tables on the tables.
• DDL: The role must have the same privileges required to create, rename, or drop the object in Snowflake. For creating a table, see the CREATE ICEBERG TABLE privileges in Configure write access to your Iceberg tables.

For more information, see Access Apache Iceberg™ tables with an external engine through Snowflake Horizon Catalog.

Data protection policies¶

An external engine that reads table files directly can’t evaluate those policies. To query a protected table, use one of the options in Enforce data protection policies on Apache Iceberg™ tables from external query engines.

For a protected externally managed table, the data served to the external engine can lag the remote catalog by up to the table’s auto-refresh interval. An unprotected externally managed table returns the latest state from the remote catalog.

Credential vending¶

Credential vending is an Iceberg REST catalog behavior. When an external engine requests vended credentials, the catalog returns a temporary cloud storage credential so the engine can read and write the table’s files directly.

When Horizon vends credentials¶

Horizon vends credentials for an externally managed Iceberg table only when all of the following are true:

• The engine sends the X-Iceberg-Access-Delegation: vended-credentials header.
• The catalog integration uses ACCESS_DELEGATION_MODE = EXTERNAL_VOLUME_CREDENTIALS.
• The database or table has an external volume that can access the table’s storage locations.
• Credential vending is enabled on the database.
• The table’s storage locations don’t overlap a Snowflake-managed Iceberg table in the same account. A location can’t be the same as, a parent of, or under a Snowflake-managed Iceberg table’s location.
• The role has privileges on the table. A role that can write receives read-write credentials. A role that can only read receives read-only credentials.

Horizon vends these credentials only on loadTable, and on createTable when stage-create is true. Credential vending on createTable applies only when you create a table in a catalog-linked database that has an external volume.

The credential covers the table location and, when they are set, write.data.path and write.metadata.path. Your cloud provider evaluates the credential at the storage layer, independently of Snowflake role-based access control. Any object under that prefix is within the credential’s scope for its lifetime.

Before you enable credential vending¶

Enable credential vending only on the databases that need it. Before you enable it, confirm the following:

• Each externally managed table has its own storage location, and that location doesn’t overlap another externally managed table or a Snowflake-managed Iceberg table.
• The external volume can access those locations, and its storage identity has only the permissions the engines need.
• Only trusted users and engines can create tables or change table locations in the remote catalog.
• Before you create or convert a Snowflake-managed Iceberg table, its location doesn’t overlap an externally managed table.

The following changes stop credential vending or change what the credential allows.

CREATE ICEBERG TABLE MANAGED_ORDERS
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = 'ANALYTICS_VOLUME'
  BASE_LOCATION = 'orders/managed_orders';

ALTER ICEBERG TABLE SALES.ORDERS CONVERT TO MANAGED;

ALTER DATABASE SALES_DB
  SET ENABLE_HORIZON_IRC_UNMANAGED_CREDENTIAL_VENDING = FALSE;

REVOKE INSERT, UPDATE, DELETE ON TABLE SALES.ORDERS FROM ROLE SPARK_WRITER;

Enable credential vending¶

To enable credential vending for a database, set the ENABLE_HORIZON_IRC_UNMANAGED_CREDENTIAL_VENDING parameter.

ALTER DATABASE <database_name>
  SET ENABLE_HORIZON_IRC_UNMANAGED_CREDENTIAL_VENDING = TRUE;

Recommendations¶

• Give each externally managed table its own storage location. Avoid broad prefixes, such as the root of an external volume or a namespace directory.
• Use separate external volumes for Snowflake-managed and externally managed Iceberg tables.
• Don’t share the storage behind an external volume with other Snowflake accounts or other systems.
• Grant each engine’s role only the privileges it needs. Use a read-only role for a read-only workload.
• Give the external volume’s storage identity read-only permissions when no engine writes.

Considerations¶

Case sensitivity and identifier resolution¶

For a catalog-linked database, Horizon IRC resolves table and namespace identifiers the same way that your remote catalog resolves them. If your catalog-linked database connects to a catalog that resolves identifiers case-sensitively, such as Apache Polaris™ or Snowflake Open Catalog, Horizon IRC also resolves them case-sensitively. If it connects to a catalog that resolves identifiers case-insensitively, such as AWS Glue or Databricks Unity Catalog, Horizon IRC resolves them case-insensitively.

For details, see Requirements for identifier resolution in a catalog-linked database.

For a standard Snowflake database, identifiers follow Snowflake identifier rules. Use the Snowflake table name.

Nested namespaces in a catalog-linked database¶

Horizon IRC uses the same namespace names as the catalog-linked database. It doesn’t show multi-level Iceberg namespaces as nested namespaces.

• With NAMESPACE_MODE = IGNORE_NESTED_NAMESPACE (the default), only first-level namespaces are available.
• With NAMESPACE_MODE = FLATTEN_NESTED_NAMESPACE, a nested namespace is available under its flattened schema name. For example, if the delimiter is _, the remote namespace accounting.tax is the schema accounting_tax. Query accounting_tax.orders, not accounting.tax.orders.
• To create a nested namespace, create the flattened name. The parent namespace must already exist.

For the parameters and examples, see NAMESPACE_MODE for a catalog-linked database.

Cross-region federation¶

Cross-region federation isn’t recommended because latency will be high. Latency and errors from the source catalog pass through Horizon Catalog to the client on every request.

Table maintenance in the source catalog¶

Configure table maintenance jobs, such as snapshot expiry, for each externally managed table in the source catalog. Without that maintenance, table metadata may grow without bound and cause performance and stability problems.

Horizon Catalog rejects a remote catalog response larger than 30 MB. If you need support for a larger response, contact Snowflake Support.

Considerations for catalog-linked databases¶

All considerations for using a catalog-linked database also apply when accessing tables through Horizon IRC. For the full list, see Considerations for using a catalog-linked database for Iceberg tables.

Limitations¶

The following limits apply when you access externally managed Iceberg tables through Horizon IRC:

• FedRAMP: This feature isn’t available on FedRAMP (Moderate) or FedRAMP (High) deployments.
• Non-Iceberg table formats: Tables in formats other than Iceberg, such as Delta, aren’t available. In a standard database, listTables returns Iceberg tables only.
• Credential vending: Credential vending is disabled by default. It isn’t supported when the catalog integration uses ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS. On a catalog-linked database, credential vending on createTable when stage-create is true requires an external volume. For more information, see Credential vending.
• registerTable: registerTable isn’t supported.
• commitTransaction: commitTransaction (multi-table commit) isn’t supported.
• Remote catalog latency and errors: Latency and errors from the source catalog pass through Horizon Catalog to the client.
• Data protection policies: Access to an externally managed Iceberg table that has a Snowflake data protection policy, such as a row access policy or a masking policy, is in public preview.