Configure a catalog integration for Apache Iceberg™ REST catalogs¶
An Apache Iceberg™ REST catalog integration lets Snowflake access Apache Iceberg™ tables managed in a remote catalog that complies with the open source Apache Iceberg REST OpenAPI specification.
Connecting to REST catalogs¶
You can connect to an Iceberg REST API that uses a public endpoint or a private network.
Public endpoint¶
To connect to an Iceberg REST API using a public endpoint, you can create a catalog integration that uses the following authentication methods:
OAuth
Bearer token or Personal Access Token (PAT)
SigV4
Private network¶
To connect to an Iceberg REST API that’s hosted in a private network, you can create a catalog integration that uses Signature Version 4 (SigV4) authentication.
Create a catalog integration¶
Create a catalog integration for your chosen authentication method using the CREATE CATALOG INTEGRATION (Apache Iceberg™ REST) command. The values you specify for the REST_CONFIG and REST_AUTHENTICATION arguments differ according to the authentication method you choose.
OAuth¶
The following example creates a REST catalog integration that uses OAuth to connect to Tabular.
CREATE OR REPLACE CATALOG INTEGRATION tabular_catalog_int
CATALOG_SOURCE = ICEBERG_REST
TABLE_FORMAT = ICEBERG
CATALOG_NAMESPACE = 'default'
REST_CONFIG = (
CATALOG_URI = 'https://api.tabular.io/ws'
WAREHOUSE = '<tabular_warehouse_name>'
)
REST_AUTHENTICATION = (
TYPE = OAUTH
OAUTH_TOKEN_URI = 'https://api.tabular.io/ws/v1/oauth/tokens'
OAUTH_CLIENT_ID = '<oauth_client_id>'
OAUTH_CLIENT_SECRET = '<oauth_secret>'
OAUTH_ALLOWED_SCOPES = ('catalog')
)
ENABLED = TRUE;
The following example creates a REST catalog integration that uses OAuth to connect to Databricks Unity Catalog.
CREATE OR REPLACE CATALOG INTEGRATION unity_catalog_int_oauth
CATALOG_SOURCE = ICEBERG_REST
TABLE_FORMAT = ICEBERG
CATALOG_NAMESPACE = 'default'
REST_CONFIG = (
CATALOG_URI = 'https://my-api/api/2.1/unity-catalog/iceberg'
)
REST_AUTHENTICATION = (
TYPE = OAUTH
OAUTH_TOKEN_URI = 'https://my-api/oidc/v1/token'
OAUTH_CLIENT_ID = '123AbC ...'
OAUTH_CLIENT_SECRET = '1365910ab ...'
OAUTH_ALLOWED_SCOPES = ('all-apis', 'sql')
)
ENABLED = TRUE;
Bearer token or PAT¶
The following example creates a REST catalog integration that uses a PAT token to connect to Databricks Unity Catalog.
CREATE OR REPLACE CATALOG INTEGRATION unity_catalog_int_pat
CATALOG_SOURCE = ICEBERG_REST
TABLE_FORMAT = ICEBERG
CATALOG_NAMESPACE = 'my_namespace'
REST_CONFIG = (
CATALOG_URI = 'https://my-api/api/2.1/unity-catalog/iceberg'
)
REST_AUTHENTICATION = (
TYPE = BEARER
BEARER_TOKEN = 'eyAbCD...eyDeF...'
)
ENABLED = TRUE;
SigV4¶
The following diagram shows how Snowflake interacts with your REST catalog server using API Gateway and SigV4 authentication.
Follow the steps in this section to use a REST API in Amazon API Gateway and Signature Version 4 (SigV4) authentication to securely connect Snowflake to an Iceberg REST catalog that isn’t publicly accessible.
Create a REST API in Amazon API Gateway¶
To connect Snowflake to your Iceberg REST catalog, you need a REST API resource in Amazon API Gateway.
If you don’t already have a REST API resource in Amazon API Gateway for your Iceberg catalog, you can create a simple REST API by importing an Iceberg catalog OpenAPI definition file or manually adding endpoints.
In the AWS Management Console, search for and select API Gateway.
Select Create API.
Select Build under REST API. To create a private REST API, select Build under REST API Private.
Select one of the following options:
To create an API by manually adding endpoints, select New API.
To create an API using an OpenAPI definition file, select Import API, then upload the file or paste the definition in the code editor.
Enter an API name and optional Description.
Note
You don’t need to enter a VPC endpoint ID when you create a private REST API.
Select Create API.
For more information about creating and developing a REST API in API Gateway, see the Amazon API Gateway Developer Guide.
Create an IAM policy and attach it to a role¶
In this step, you create an AWS IAM role that Snowflake can use to connect to API Gateway. You attach a policy to the role that grants permission to call your API.
In the AWS Management Console, search for and select IAM.
From the left-hand navigation pane, select Policies.
Select Create policy and then select JSON for the Policy editor.
Replace the empty policy with a policy that has permission to invoke your API methods. For example, the following general policy allows the invoke action for all API Gateway resources in an AWS account.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "execute-api:Invoke" ], "Resource": "arn:aws:execute-api:*:<aws_account_id>:*" } ] }
Important
As a best practice, use a policy that grants the minimum required privileges for your use case. For additional guidance and example policies, see Control access to an API with IAM permissions.
Select Next.
Enter a Policy name (for example,
snowflake_access
) and an optional Description.Select Create policy.
From the left-hand navigation pane in the IAM dashboard, select Roles.
Select a role to attach the policy to. When you create a catalog integration, you specify this role. If you don’t have a role, create a new role.
On the role Summary page in the Permissions tab, select Add permissions » Attach policies.
Search for and check the box next to the policy that you created for API Gateway, then select Add permissions.
On the role Summary page, copy the role ARN. You specify this ARN when you create a catalog integration.
Attach an API Gateway resource policy (private APIs only)¶
If your REST API is private, you must attach an Amazon API Gateway resource policy to your API. The resource policy allows Snowflake to call your API from the Amazon Virtual Private Cloud (VPC) in which your Snowflake account is located.
In Snowflake, call the SYSTEM$GET_SNOWFLAKE_PLATFORM_INFO function to retrieve the ID for the VPC in which your Snowflake account is located. Copy the VPC ID from the function output.
SELECT SYSTEM$GET_SNOWFLAKE_PLATFORM_INFO();
Output:
{"snowflake-vpc-id":["vpc-c1c234a5"]}
Follow the instructions in Attaching API Gateway resource policies to attach a resource policy to your REST API.
Paste and modify the following example policy.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Principal": "*", "Action": "execute-api:Invoke", "Resource": "<api_gateway_arn>", "Condition": { "StringNotEquals": { "aws:sourceVpc": "<snowflake_vpc_id>" } } }, { "Effect": "Allow", "Principal": { "AWS": "arn:aws:sts::123456789012:assumed-role/<my_api_permissions_role_name>/snowflake" }, "Action": "execute-api:Invoke", "Resource": "<api_gateway_arn>/*/*/*", "Condition": { "StringEquals": { "aws:sourceVpc": "<snowflake_vpc_id>" } } } ] }
The first statement in the policy denies all requests that don’t originate from the Snowflake VPC. The second statement allows the invoke action (for all methods) from requests originating from the Snowflake VPC that use the assumed-role session principal.
To learn more about API Gateway resource policies, see:
Retrieve the endpoint URL¶
Retrieve your REST API endpoint URL (or invoke URL). Your API must be deployed to a stage before you can retrieve the endpoint URL.
In the Amazon API Gateway console, select your REST API.
In the left-hand navigation pane, select Stages.
Under Stage details, copy the Invoke URL.
You specify the endpoint URL when you create a catalog integration.
Create a catalog integration for SigV4¶
After you have a REST API in Amazon API Gateway and have completed the initial steps to control access to your API using IAM permissions, you can create a catalog integration in Snowflake.
To view the command syntax and parameter descriptions, see CREATE CATALOG INTEGRATION (Apache Iceberg™ REST).
Public REST API
To create a catalog integration for a public REST API, specify ICEBERG_REST
as the CATALOG_SOURCE
and use SIGV4
authentication.
Include details such as your API endpoint URL and IAM role ARN.
CREATE OR REPLACE CATALOG INTEGRATION my_rest_catalog_integration
CATALOG_SOURCE = ICEBERG_REST
TABLE_FORMAT = ICEBERG
CATALOG_NAMESPACE = 'my_namespace'
REST_CONFIG = (
CATALOG_URI = 'https://asdlkfjwoalk-execute-api.us-west-2-amazonaws.com/MyApiStage'
)
REST_AUTHENTICATION = (
TYPE = SIGV4
SIGV4_IAM_ROLE = 'arn:aws:iam::123456789012:role/my_api_permissions_role'
)
ENABLED = TRUE;
Private REST API
To create a catalog integration for a private REST API, you must set the CATALOG_API_TYPE
parameter to AWS_PRIVATE_API_GATEWAY
.
CREATE OR REPLACE CATALOG INTEGRATION my_rest_catalog_integration
CATALOG_SOURCE = ICEBERG_REST
TABLE_FORMAT = ICEBERG
CATALOG_NAMESPACE = 'my_namespace'
REST_CONFIG = (
CATALOG_URI = 'https://asdlkfjwoalk-execute-api.us-west-2-amazonaws.com/MyApiStage'
)
REST_AUTHENTICATION = (
TYPE = SIGV4
CATALOG_API_TYPE = AWS_PRIVATE_API_GATEWAY
SIGV4_IAM_ROLE = 'arn:aws:iam::123456789012:role/my_api_permissions_role'
)
ENABLED = TRUE;
Configure the trust relationship in IAM¶
Next, you must retrieve information about the AWS IAM user and the external ID that were created for your Snowflake account when you created the catalog integration, and configure the trust relationship for your IAM role.
In Snowflake, call the DESCRIBE CATALOG INTEGRATION command:
DESCRIBE CATALOG INTEGRATION my_rest_catalog_integration;
Record the following values:
Value
Description
API_AWS_IAM_USER_ARN
The AWS IAM user created for your Snowflake account, for example,
arn:aws:iam::123456789001:user/abc1-b-self1234
. Snowflake provisions a single IAM user for your entire Snowflake account.API_AWS_EXTERNAL_ID
The external ID that’s needed to establish a trust relationship.
In the AWS Management Console, search for and select IAM.
From the left-hand navigation pane, select Roles.
Select the IAM role that you created for your catalog integration.
Select the Trust relationships tab.
Select Edit trust policy.
Modify the policy document with the values that you recorded.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "<api_aws_iam_user_arn>" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "<api_aws_external_id>" } } } ] }
Select Update policy to save your changes.