Configure an Iceberg REST catalog integration for Unity with outbound private connectivity¶

Complete the steps in this topic to configure a catalog integration to connect to Databricks Unity Catalog through a private IP address instead of over the public internet.

With this configuration, you can use the catalog integration to connect to the Unity Catalog through a private IP address instead of over the public internet.

Note

This feature is only supported for Snowflake accounts that are hosted on AWS.

Prerequisites¶

Your Snowflake account and Unity Catalog must be located in the same AWS region.

Step 1: Join the private preview¶

To enroll in the private preview, contact your account team. To use this feature, you must enroll in the private preview.

Step 2: Provision a private connectivity endpoint¶

In this section, you provision a private connectivity endpoint in the Snowflake VPC or VNet. This enables Snowflake to connect to Unity by using private connectivity.

You only need to provision one private connectivity endpoint, which is an AWS resource. Unity requires just one private connectivity endpoint to access everything managed by the Unity Data Catalog in the same region.

To create a private connectivity endpoint, call the SYSTEM$PROVISION_PRIVATELINK_ENDPOINT system function:
```
USE ROLE ACCOUNTADMIN;

SELECT SYSTEM$PROVISION_PRIVATELINK_ENDPOINT(
  '<vpc_endpoint_service_id>',
  '<databricks_workspace_URL>'
);
```
Copy
Where:
<vpc_endpoint_service_id> is the PrivateLink VPC endpoint service ID for the Databricks Unity Catalog. For example: com.amazonaws.vpce.us-west-2.vpce-svc-0129f463fcfbc46c5. For the list of these endpoint service IDs, see PrivateLink VPC endpoint services in the Databricks documentation. For more information about PrivateLink at Databricks, see Enable private connectivity using AWS PrivateLink in the Databricks documentation.

<databricks_workspace_URL> is the Databricks workspace URL.
Note

If you have multiple Databricks workspaces in the same AWS region, you can use a wildcard with your Databricks workspace URL.
To verify the endpoint status, run the following query:
```
SELECT SYSTEM$GET_PRIVATELINK_ENDPOINTS_INFO();
```
Copy
The endpoint is ready to use when the status changes from pending to available.

Step 3: Register the Snowflake VPC endpoint in Databricks¶

In this section, you register the Snowflake VPC endpoint in Databricks to accept the traffic coming from the VPC endpoint.

Complete configuration steps in Databricks¶

Before you register the Snowflake VPC endpoint, make sure you configure the following in Databricks:

Your workspace must be located in a customer-managed VPC.
Your Databricks account must be in the enterprise subscription.
You need to set up private access configuration.

For more information, see Configure private connectivity to Databricks in the Databricks documentation.

Register the Snowflake VPC endpoint¶

To register the VPC endpoint, follow these steps:

In Snowflake, run the following system function and copy the value for snowflake_endpoint_name in the response:
```
SELECT SYSTEM$GET_PRIVATELINK_ENDPOINTS_INFO();
```
Copy
For example, the output to copy looks like vpce-11111aaaa11aaaa11. This value is the VPC endpoint ID in your Snowflake account.
In Databricks, register the Snowflake VPC endpoint ID by specifying the VPC endpoint ID value you copied in the previous step. For more instructions, see Manage VPC endpoint registrations in the Databricks documentation.
In Databricks, add a private access setting and specify the VPC endpoint that you registered in the previous step. For instructions, see Manage private access settings in the Databricks documentation.

Step 4: Configure a catalog integration for Unity Iceberg REST¶

In this section, you configure a catalog integration for the Unity Iceberg REST endpoint.

To configure this catalog integration, follow the steps in Configure a catalog integration for Unity Catalog.

Important

To use outbound private connectivity, you must specify CATALOG_API_TYPE = PRIVATE as one of the restConfigParams when you create the integration.

CREATE OR REPLACE CATALOG INTEGRATION unity_catalog_int_private_connectivity_pat
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT = ICEBERG
  CATALOG_NAMESPACE = 'my_namespace'
  REST_CONFIG = (
    CATALOG_URI = 'https://my-api/api/2.1/unity-catalog/iceberg-rest'
    CATALOG_NAME = '<catalog_name>'
    CATALOG_API_TYPE = PRIVATE
  )
  REST_AUTHENTICATION = (
    TYPE = BEARER
    BEARER_TOKEN = 'eyAbCD...eyDeF...'
  )
  ENABLED = TRUE;

Copy

Step 5: Configure an external volume with outbound private connectivity¶

In this section, you set up outbound private connectivity to an external volume for AWS.

Snowflake must have access to your storage location through outbound private connectivity for the following reasons:

Snowflake needs to write data to the storage.
This access allows Unity to read metadata from and write metadata to the storage location through AWS. PrivateLink.

To configure this external volume, follow the instructions in Syntax updates for private connectivity to external volumes for AWS.

Next steps¶

After you configure a catalog integration for AWS Glue Iceberg REST, you can create an Iceberg table.

Note

When you create the table, specify a catalog integration that is configured with outbound private connectivity.