Using Cortex Code CLI with other data platforms¶

Capabilities¶

AWS side prerequisites¶

• AWS CLI installed and configured. If it is not installed, Cortex Code provides installation instructions.

• An AWS profile with sufficient permissions to create Glue databases, crawlers, and IAM roles, and to run Athena queries. See the minimum operator policy below.

• Athena engine version 3 (Trino-based) workgroup if you are converting data to Iceberg format. Cortex Code checks the workgroup version and prompts you to upgrade if needed.

The following IAM policy covers the minimum permissions needed to run this skill:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueSetup",
      "Effect": "Allow",
      "Action": [
        "glue:CreateDatabase", "glue:GetDatabase", "glue:GetDatabases",
        "glue:CreateCrawler", "glue:StartCrawler", "glue:GetCrawler",
        "glue:DeleteCrawler",
        "glue:GetTable", "glue:GetTables", "glue:UpdateTable"
      ],
      "Resource": "*"
    },
    {
      "Sid": "IAMRoleCreation",
      "Effect": "Allow",
      "Action": [
        "iam:CreateRole", "iam:AttachRolePolicy", "iam:PutRolePolicy",
        "iam:GetRole", "iam:ListAttachedRolePolicies", "iam:PassRole"
      ],
      "Resource": "arn:aws:iam::<account-id>:role/<crawler-role-name>"
    },
    {
      "Sid": "S3Access",
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets", "s3:GetBucketLocation",
        "s3:ListBucket", "s3:GetObject",
        "s3:PutObject", "s3:DeleteObject", "s3:CreateBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<source-bucket>",
        "arn:aws:s3:::<source-bucket>/*",
        "arn:aws:s3:::<athena-results-bucket>",
        "arn:aws:s3:::<athena-results-bucket>/*"
      ]
    },
    {
      "Sid": "AthenaExecution",
      "Effect": "Allow",
      "Action": [
        "athena:StartQueryExecution", "athena:GetQueryExecution",
        "athena:GetWorkGroup", "athena:ListWorkGroups"
      ],
      "Resource": "*"
    },
    {
      "Sid": "LakeFormation",
      "Effect": "Allow",
      "Action": [
        "lakeformation:GetDataLakeSettings",
        "lakeformation:GrantPermissions",
        "lakeformation:ListResources"
      ],
      "Resource": "*"
    }
  ]
}

If you already have a crawler IAM role, omit the IAMRoleCreation block. If Lake Formation is not enabled in your account, omit the LakeFormation block.

Getting started¶

Start a Cortex Code session. Cortex Code asks for your AWS CLI profile and region before running any commands.

If your data is already in Iceberg format in S3 and you want to register it in Glue:

Register my Iceberg tables in AWS Glue Data Catalog

If your data is in parquet, CSV, or JSON and needs converting:

Set up a Glue database and convert my parquet data to Iceberg using Athena

To run individual phases:

Check my AWS credentials and list my S3 buckets

Create a Glue crawler for s3://my-bucket/data/ and run it

Validate the schemas for the tables my Glue crawler discovered

Convert my Glue external tables to Iceberg using Athena CTAS

At the end of AWS setup, Cortex Code asks whether you want to continue with Snowflake catalog integration. If yes, it passes all collected variables (account ID, region, Glue database name, IAM role ARN, table names) directly to the catalog integration skill.

Troubleshooting¶

AWS credentials expired

Symptom: Commands fail with ExpiredTokenException or ExpiredToken.

Solution: Run aws sso login --profile <profile> to refresh your session, or run aws configure --profile <profile> to reconfigure credentials.

Lake Formation permission error on crawler

Symptom: The Glue crawler fails with Insufficient Lake Formation permission(s): Required Describe on <database>.

Solution: Lake Formation is active in your account. The crawler IAM role needs explicit LF grants on the Glue database. Use the prompt My Glue crawler is failing with a Lake Formation permission error and Cortex Code detects whether LF is in restrictive or IAM compatibility mode and generates the correct grant-permissions commands.

Tables not discovered correctly after crawling Iceberg data

Symptom: The crawler creates entries for metadata files (metadata/, *.avro) instead of the actual Iceberg tables, or table names don’t match expectations.

Solution: The standard S3 Glue crawler does not natively recognize Iceberg format. Use the prompt Register my existing Iceberg tables in Glue to bypass the crawler and register tables directly with the correct table_type = ICEBERG parameter and metadata location.

Athena CTAS fails with HIVE_BAD_DATA

Symptom: Parquet-to-Iceberg conversion fails with a type mismatch error.

Solution: The source table has column types that Athena cannot implicitly cast. Cortex Code re-generates the CTAS statement with explicit CAST() expressions for the affected columns. You can also ask Switch to Glue Spark for the conversion to use the add_files approach instead of Athena.

Athena backtick syntax error

Symptom: CTAS or SELECT queries fail with SYNTAX_ERROR: backquoted identifiers are not supported.

Solution: Athena’s Trino engine (v3) uses double quotes for identifiers, not backticks. Cortex Code automatically uses the correct quoting in generated SQL.

Duplicate partition column error

Symptom: Athena queries on a partitioned table fail with a duplicate column error after crawling.

Solution: This occurs when Snowflake-exported parquet files include the partition column both inside the file and in the Hive directory path (column=value/). Use the prompt Fix duplicate partition columns in my Glue table and Cortex Code runs aws glue update-table to remove the duplicate from StorageDescriptor.Columns.

Capabilities¶

Snowflake prerequisites¶

Snowflake:

• ACCOUNTADMIN role, or explicit grants for CREATE INTEGRATION and CREATE EXTERNAL VOLUME on the account.

• Business Critical Edition or higher if you are using PrivateLink connectivity.

AWS:

• An AWS account with Glue Data Catalog configured and Iceberg tables registered.

• An IAM role that Snowflake can assume to access Glue and S3.

• Lake Formation enabled and configured if you are using catalog-vended credentials (recommended).

Getting started¶

Start a Cortex Code session and use natural language to trigger the skill:

Set up a catalog integration for my AWS Glue Data Catalog

Cortex Code walks you through the setup one step at a time, collecting your AWS account ID, region, IAM role ARN, access delegation mode, and connectivity type before generating the SQL.

To verify an existing integration:

Verify my Glue catalog integration

To troubleshoot a failing integration:

My Glue catalog integration returns an error — help me diagnose it

Authentication¶

The Glue catalog integration uses AWS SigV4 authentication. Snowflake assumes an IAM role in your AWS account to interact with Glue and S3.

During setup, Cortex Code generates the CREATE CATALOG INTEGRATION SQL with the role ARN you provide, then retrieves the Snowflake IAM user ARN and external ID. You add those values to your IAM role’s trust policy to authorize the connection.

Troubleshooting¶

Trust relationship not configured

Symptom: Verification returns an Access denied or InvalidClientTokenId error.

Solution: Run DESC CATALOG INTEGRATION <name> to retrieve the Snowflake IAM user ARN and external ID, then add them to your IAM role’s trust policy in the AWS console.

External ID mismatch

Symptom: Verification fails with an authentication error even though the trust policy exists.

Solution: Re-run DESC CATALOG INTEGRATION <name> to get the current external ID and update the trust policy condition to match exactly.

Lake Formation access denied

Symptom: Tables are visible but queries return a 403 Forbidden or Lake Formation permission error.

Solution: Grant the IAM role SELECT permission on the Glue database and tables in the Lake Formation console.

PrivateLink endpoint not available

Symptom: CREATE CATALOG INTEGRATION with CATALOG_API_TYPE = AWS_PRIVATE_GLUE fails with a connectivity error.

Solution: Run SELECT SYSTEM$PROVISION_PRIVATELINK_ENDPOINT('com.amazonaws.<region>.glue', 'glue.<region>.amazonaws.com') to provision the endpoint, wait for it to become available, then retry.

Cross-region PrivateLink not supported

Symptom: PrivateLink provisioning fails when your Snowflake account and Glue Data Catalog are in different regions.

Solution: Use public connectivity, or ensure your Snowflake account and Glue Data Catalog are in the same AWS region. AWS does not support cross-region PrivateLink for the Glue service.