Set up Openflow¶

This topic describes the steps to set up Openflow.

Setting up Openflow involves the following steps:

Prerequisites¶

The prerequisites to be completed on your Snowflake and AWS accounts are as follows:

Snowflake account¶

  • If you’ve never used image repositories in your Snowflake account, you need to set the following to make sure the deployment service can pull the Openflow images from Snowflake.

    use role accountadmin;
    
    CREATE DATABASE IF NOT EXISTS OPENFLOW;
    USE OPENFLOW;
    CREATE SCHEMA IF NOT EXISTS OPENFLOW;
    USE SCHEMA OPENFLOW;
    CREATE IMAGE REPOSITORY IF NOT EXISTS OPENFLOW;
    grant usage on database OPENFLOW to role public;
    grant usage on schema OPENFLOW to role public;
    grant read on image repository OPENFLOW.OPENFLOW.OPENFLOW to role public;
    
    Copy
  • Setting up Openflow requires defining new privileges at the Snowflake Account level. The new privileges are assigned to the ACCOUNTADMIN role as part of the default set of privileges. ACCOUNTADMIN will automatically have the following two privileges and will be able to grant them to a role of their choosing for the Openflow admin role, denoted as $OPENFLOW_ADMIN_ROLE in the following code:

    use role accountadmin;
    
    grant create openflow data plane integration to role $OPENFLOW_ADMIN_ROLE;
    grant create openflow runtime integration to role $OPENFLOW_ADMIN_ROLE;
    
    Copy

Deployment integration privileges¶

The deployment integration object represents a set of resources provisioned to deploy one or more Snowflake Openflow runtimes. For organizations bringing their own cloud resources, the deployment integration object represents a managed Kubernetes cluster along with its associated nodes.

Users with the CREATE DATA PLANE INTEGRATION privilege on the Snowflake account can create and delete the deployment integration objects.

Additional privileges can be defined on deployment integration objects directly to support differentiation of access.

You can grant the following privileges on a deployment integration object:

  • OWNERSHIP: Enables full control over deployment actions objects, including deletion of the deployment.

  • USAGE: Enables creation of runtime child objects.

Runtime privileges¶

The runtime object represents a cluster of one or more Snowflake Openflow runtime servers, provisioned to run flow definitions. For Kubernetes deployments, the runtime object represents a stateful set of Snowflake Openflow Runtime containers deployed in a namespace, along with supporting components.

Users with the OWNERSHIP privilege on the parent deployment integration object and the CREATE RUNTIME INTEGRATION account-level privilege can create runtime integration objects. Additional privileges can be defined on runtime integration objects directly to support differentiation of access.

You can grant the following privileges on a runtime integration object:

  • OWNERSHIP: Enables full control over runtime actions, including deletion of the associated runtime and modification of runtime flow definitions.

  • USAGE: Enables read access to the deployed Runtime for observing health and status, without making any changes.

AWS account¶

Ensure the following on your AWS account:

  • You have an AWS account with permissions required to create a CloudFormation stack.

  • You have an AWS administrator in your organization with the ability to execute CloudFormation script to set up EKS inside a new VPC (created by CloudFormation) or an existing VPC.

Note

To learn about how the Openflow installation happens in your AWS account and the permissions that are configured by the CloudFormation template, see Installation process.

Accept the Openflow terms of service¶

This step is only required once for your organization.

  1. Sign in to Snowflake as a user with the ORGADMIN role.

  2. Navigate to Data » Openflow.

  3. Accept Openflow terms of services.

Create a deployment in your cloud¶

Important

You cannot sign in to Openflow as an ACCOUNTADMIN. Switch to a role that your ACCOUNTADMIN assigned for using Openflow. For more information, see Prerequisites.

  1. Sign in to Snowflake with a role that your ACCOUNTADMIN assigned for using Openflow.

  2. Navigate to Data » Openflow.

  3. Select Launch Openflow.

  4. In the Openflow UI, select Create a deployment. The Deployments tab opens.

  5. Select Create a deployment. The Creating a deployment wizard opens.

  6. In the Prerequisites step, ensure that you meet all the requirements. Select Next.

  7. In the Deployment location step, select Amazon Web Services as the deployment location. Enter a name for your deployment. Select Next.

  8. In the Configuration step, you can select one of the following configuration options:

    • Fully managed VPC: Choose this option if you want your VPC to be managed by Snowflake

    • Bring your own VPC: Choose this option if you want to use an existing VPC.

  9. Select Create Deployment.

  10. Once your deployment is configured, a dialog box appears that lets you download the CloudFormation template to complete the setup process in your AWS account. Download this template and navigate to your AWS account.

  11. In your AWS account, create a new CloudFormation Stack using the template. After the Openflow deployment agent’s EC2 instance is created, it completes the rest of the Installation process using infrastructure as code scripts.

  12. Find the NAT gateway IP address.

  13. Navigate to your Snowflake account.

  14. [Optional] This step is required only if you’re using network policies to control access to Snowflake. Create a network policy. A network policy is a set of rules that control which IP addresses can access your Snowflake account. Replace {$NAT_GATEWAY_IP} in the following code snippet with the NAT gateway IP address. NAT gateway is responsible for Openflow egress for both DPA and EKS. Both DPA and EKS run in the Private Subnet 1 of the installation.

    USE ROLE ACCOUNTADMIN;
    CREATE NETWORK RULE allow_openflow_deployment
    MODE = INGRESS
    TYPE = IPV4
    VALUE_LIST = ('{$NAT_GATEWAY_IP}/32');
    
    CREATE NETWORK POLICY openflow_deployment ALLOWED_NETWORK_RULE_LIST = (allow_openflow_deployment);
    
    ALTER ACCOUNT SET NETWORK_POLICY = openflow_deployment;
    
    Copy
  15. Create an event table. An event table is designed to store and manage event data, such as user actions, system logs, and transactions. You can choose to use an Openflow-specific event table or an account-specific event table.

    • Create an Openflow-specific event table

      USE ROLE accountadmin;
      GRANT create event table on schema OPENFLOW.OPENFLOW to role $ROLE_OF_DEPLOYMENT_OWNER;
      
      USE ROLE $ROLE_OF_DEPLOYMENT_OWNER;
      CREATE event table if not exists openflow.openflow.openflow_events;
      
      -- Find the Data Plane Integrations
      SHOW openflow data plane integrations;
      
      ALTER openflow data plane integration
      $OPENFLOW_deployment_UUID
      SET event_table = 'openflow.openflow.openflow_events';
      
      Copy
    • Create an account-specific event table

      USE database openflow;
      CREATE schema if not exists openflow.telemetry;
      CREATE event table if not exists openflow.telemetry.events;
      ALTER ACCOUNT SET EVENT_TABLE = openflow.telemetry.events;
      
      Copy
  16. Navigate to Openflow UI. Creating a deployment takes about 45 minutes on AWS. Once it’s created, you can view your deployment in the Deployments tab of Openflow UI with its state marked as Active.

Create a runtime environment in your cloud¶

  1. In Openflow Control Plane, select Create a runtime. The Create Runtime dialog box appears.

  2. From the Deployment drop-down list, choose the deployment in which you want to create a runtime.

  3. Enter a name for your runtime.

  4. Choose a node type from the Node type drop-down list. This specifies the size of your nodes.

  5. In the Min/Max node range selector, select a range. The minimum value specifies the number of nodes that get created when the runtime starts and the maximum value specifies the number of nodes that the runtime can scale up to, in the event of high data volume or CPU load.

  6. Select Create. The runtime takes a couple of minutes to get created.

Once created, you can view your runtime by navigating to the Runtimes tab of the Openflow control plane. Click the runtime to open the Openflow canvas.

Next step¶

Deploy a connector in a runtime. For a list of connectors available in Openflow, see Openflow connectors.

Installation process¶

Between the CloudFormation stack and the Openflow Agent, there are several coordinated steps that the BYOC deployment installation process manages. The goal is to separate responsibilities between a cold-start that gives organizations an easy way to provide inputs to their BYOC deployment (solved via CloudFormation), and the configuration of the deployment and its core software components that will need to change over time (solved by the Openflow Agent).

Deployment agent: The deployment Agent facilitates the creation of the Openflow deployment infrastructure and installation of the deployment software components including the deployment service. The deployment agent authenticates with Snowflake System Image Registry to obtain Openflow container images.

The steps are as follows:

  1. The CloudFormation template creates the following and configures with the AWS permissions mentioned in Configured AWS permissions:

    • One VPC with two public subnets and two private subnets. Public subnets host the AWS Network Load Balancer (created later). Private Subnets host the EKS cluster and all of the EC2 instances backing the NodeGroups. Openflow runtimes run within a private subnet.

    • Internet Gateway for egress from the VPC

    • NAT Gateway for egress from the private subnets

    • AWS Secrets Manager entry for the OIDC configuration input by the user

    • IAM role and instance profile for the Openflow Agent to use from its EC2 instance

    • An EC2 instance for Openflow deploymnent agent, complete with a UserData script to automatically run the initialization process. This script sets environment variables for the Openflow deploymnent agent to use, derived from the input CloudFormation parameters.

    • EC2 Instance Connect endpoint for the Openflow deploymnent agent to upgrade the deployment when needed.

    • S3 Bucket that stores the Terraform state, logs, and outputs for the Openflow Agent

  2. The Openflow deployment agent creates the following:

    • An EKS cluster containing:

      • Node groups

      • Autoscaling groups

      • VPC CNI add-on

      • EBS CSI add-on

    • Secrets manager records for PostgreSQL, OAuth credentials, and so on.

    • IAM policies and roles for various K8s service accounts to retrieve their secrets from AWS Secrets Manager.

    • K8s components

      • Namespaces

      • Cluster autoscaler

      • EBS CSI expandable storage

      • AWS Load Balancer Controller, which creates the publicly accessible Network Load Balancer

      • Let’s Encrypt certificate issuer

      • Nginx Ingress, configured for Let’s Encrypt

      • Metrics Server

      • Certificate manager from Jetstack

      • External secrets operator

      • Service accounts for Temporal, deployment service, and OIDC

      • Secrets stores for Temporal, deployment service, and OIDC

      • External secrets for Temporal and deployment service. The external secret for OIDC is created and managed by the runtime operator.

      • PostgreSQL

      • Temporal

      • Self-signed certificate issuer and ingress configuration for communications between runtime nodes

      • Openflow runtime operator

      • Openflow deployment service

By default, all AWS accounts have a quota of five Elastic IP addresses per region, because public (IPv4) internet addresses are a scarce public resource. Snowflake strongly recommends that you use Elastic IP addresses primarily for their ability to remap the address to another instance in the case of instance failure, and to use DNS hostnames for all other inter-node communication.

Configured AWS permissions¶

This section lists the AWS permissions configured by Openflow BYOC stack based on the roles.

Note

{key} represents the deployment key that uniquely identifies cloud resources created and managed by Openflow for a particular deployment.

Administrative user

cloudformation and all of the following permissions.

IAM Role: openflow-agent-role-{key}

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "autoscaling:DescribeTags",
               "ec2:DescribeImages",
               "ec2:DescribeInstances",
               "ec2:DescribeLaunchTemplates",
               "ec2:DescribeLaunchTemplateVersions",
               "ec2:DescribeNetworkInterfaces",
               "ec2:DescribeSecurityGroups",
               "ec2:DescribeSubnets",
               "ec2:DescribeTags",
               "ec2:DescribeVolumes",
               "ec2:DescribeVpcs",
               "ec2:DescribeVpcAttribute",
               "iam:GetRole",
               "iam:GetOpenIDConnectProvider",
               "ecr:GetAuthorizationToken",
               "ec2:RunInstances",
               "ec2:CreateLaunchTemplate",
               "ec2:CreateSecurityGroup",
               "ec2:CreateTags",
               "ec2:DeleteTags"
            ],
            "Resource": "*",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "aws:ResourceTag/Name": [
                        "{key}-oidc-provider"
                  ]
               }
            },
            "Action": [
               "iam:CreateOpenIDConnectProvider",
               "iam:DeleteOpenIDConnectProvider",
               "iam:TagOpenIDConnectProvider"
            ],
            "Resource": "arn:aws:iam::{Account_ID}:oidc-provider/oidc.eks.{Region}.amazonaws.com/id/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "iam:DeletePolicy",
               "iam:CreatePolicy",
               "iam:GetPolicy",
               "iam:GetPolicyVersion",
               "iam:ListPolicyVersions"
            ],
            "Resource": [
               "arn:aws:iam::{Account_ID}:policy/dp-service-role-policy-{key}",
               "arn:aws:iam::{Account_ID}:policy/oauth2-role-policy-{key}",
               "arn:aws:iam::{Account_ID}:policy/temporal-service-role-policy-{key}",
               "arn:aws:iam::{Account_ID}:policy/oidc-service-role-policy-{key}",
               "arn:aws:iam::{Account_ID}:policy/dps-temporal-role-policy-{key}"
               "arn:aws:iam::{Account_ID}:policy/dps-postgres-role-policy-{key}"
            ],
            "Effect": "Allow"
      },
      {
            "Action": [
               "iam:UpdateAssumeRolePolicy",
               "iam:PutRolePolicy",
               "iam:ListInstanceProfilesForRole",
               "iam:ListAttachedRolePolicies",
               "iam:ListRolePolicies",
               "iam:GetRolePolicy",
               "iam:CreateRole",
               "iam:AttachRolePolicy",
               "iam:DeleteRole",
               "iam:DeleteRolePolicy",
               "iam:DetachRolePolicy",
               "iam:TagRole"
            ],
            "Resource": [
               "arn:aws:iam::{Account_ID}:role/openflow-agent-role-{key}",
               "arn:aws:iam::{Account_ID}:role/{key}-*",
               "arn:aws:iam::{Account_ID}:role/dps-temporal-role-{key}",
               "arn:aws:iam::{Account_ID}:role/dps-postgres-role-{key}",
               "arn:aws:iam::{Account_ID}:role/dp-service-role-{key}",
               "arn:aws:iam::{Account_ID}:role/oauth2-role-{key}",
               "arn:aws:iam::{Account_ID}:role/oidc-service-role-{key}"
            ],
            "Effect": "Allow"
      },
      {
            "Action": [
               "autoscaling:CreateOrUpdateTags",
               "autoscaling:DeleteTags"
            ],
            "Resource": "arn:aws:autoscaling:{Region}:{Account_ID}:autoScalingGroup:*:autoScalingGroupName/eks-{key}-*",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "aws:ResourceTag/Name": [
                        "{key}-EC2SecurityGroup-*",
                        "k8s-traffic-{key}-*",
                        "eks-cluster-sg-{key}-*",
                        "{key}-cluster-sg",
                        "postgres-{key}-sg"
                  ]
               }
            },
            "Action": [
               "ec2:AuthorizeSecurityGroupEgress",
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:RevokeSecurityGroupEgress",
               "ec2:DeleteSecurityGroup",
               "ec2:CreateTags",
               "ec2:DeleteTags",
               "ec2:CreateNetworkInterface",
               "ec2:DeleteNetworkInterface"
            ],
            "Resource": "arn:aws:ec2:{Region}:{Account_ID}:security-group/*",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "aws:ResourceTag/elbv2.k8s.aws/cluster": "{key}"
               }
            },
            "Action": [
               "ec2:AuthorizeSecurityGroupEgress",
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:RevokeSecurityGroupEgress",
               "ec2:DeleteSecurityGroup",
               "ec2:CreateTags",
               "ec2:DeleteTags",
               "ec2:CreateNetworkInterface",
               "ec2:DeleteNetworkInterface"
            ],
            "Resource": "arn:aws:ec2:{Region}:{Account_ID}:security-group/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "ec2:CreateSecurityGroup"
            ],
            "Resource": "arn:aws:ec2:{Region}:{Account_ID}:vpc/vpc-018d2da0fde903de4",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/Name": "openflow-agent-{key}"
               }
            },
            "Action": [
               "ec2:AttachNetworkInterface"
            ],
            "Resource": "arn:aws:ec2:{Region}:{Account_ID}:instance/*",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "aws:ResourceTag/Name": "{key}-*-group"
               }
            },
            "Action": [
               "ec2:DeleteLaunchTemplate"
            ],
            "Resource": "arn:aws:ec2:{Region}:{Account_ID}:launch-template/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "eks:CreateCluster",
               "eks:CreateAccessEntry",
               "eks:CreateAddon",
               "eks:CreateNodegroup",
               "eks:DeleteCluster",
               "eks:DescribeCluster",
               "eks:ListClusters",
               "eks:ListNodeGroups",
               "eks:DescribeUpdate",
               "eks:UpdateClusterConfig",
               "eks:TagResource"
            ],
            "Resource": "arn:aws:eks:{Region}:{Account_ID}:cluster/{key}",
            "Effect": "Allow"
      },
      {
            "Action": [
               "eks:DescribeAddon",
               "eks:DescribeAddonVersions",
               "eks:UpdateAddon",
               "eks:DeleteAddon",
               "eks:DescribeUpdate"
            ],
            "Resource": "arn:aws:eks:{Region}:{Account_ID}:addon/{key}/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "eks:DeleteNodegroup",
               "eks:DescribeNodegroup",
               "eks:ListNodegroups",
               "eks:UpdateNodegroupConfig",
               "eks:TagResource",
               "eks:DescribeUpdate"
            ],
            "Resource": "arn:aws:eks:{Region}:{Account_ID}:nodegroup/{key}/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "s3:CreateBucket",
               "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::byoc-tf-state-{key}",
            "Effect": "Allow"
      },
      {
            "Action": [
               "s3:DeleteObject",
               "s3:GetObject",
               "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::byoc-tf-state-{key}/*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "secretsmanager:CreateSecret",
               "secretsmanager:DeleteSecret",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:GetSecretValue",
               "secretsmanager:PutSecretValue",
               "secretsmanager:UpdateSecretVersionStage"
            ],
            "Resource": "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:*-{key}*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "ecr:BatchCheckLayerAvailability",
               "ecr:BatchGetImage",
               "ecr:DescribeImages",
               "ecr:DescribeRepositories",
               "ecr:GetDownloadUrlForLayer",
               "ecr:ListImages"
            ],
            "Resource": "arn:aws:ecr:{Region}:{Account_ID}:*",
            "Effect": "Allow"
      },
      {
            "Action": [
               "ecr:CreateRepository",
               "ecr:CompleteLayerUpload",
               "ecr:InitiateLayerUpload",
               "ecr:PutImage",
               "ecr:UploadLayerPart"
            ],
            "Resource": "arn:aws:ecr:{Region}:{Account_ID}:repository/snowflake-openflow/*",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "iam:AWSServiceName": "eks.amazonaws.com"
               }
            },
            "Action": [
               "iam:CreateServiceLinkedRole"
            ],
            "Resource": "arn:aws:iam::*:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS",
            "Effect": "Allow"
      },
      {
            "Condition": {
               "StringLike": {
                  "iam:AWSServiceName": "eks-nodegroup.amazonaws.com"
               }
            },
            "Action": [
               "iam:CreateServiceLinkedRole"
            ],
            "Resource": "arn:aws:iam::*:role/aws-service-role/eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup",
            "Effect": "Allow"
      },
      {
            "Action": [
               "eks:AssociateAccessPolicy",
               "eks:ListAssociatedAccessPolicies",
               "eks:DisassociateAccessPolicy"
            ],
            "Resource": "arn:aws:eks:{Region}:{Account_ID}:access-entry/{key}/*",
            "Effect": "Allow"
      },
      {
            "Action": "iam:PassRole",
            "Resource": "*",
            "Effect": "Allow"
      }
   ]
}
Copy

IAM Role: {key}-cluster-ServiceRole

AWS-managed policies:

  • AmazonEKSClusterPolicy

  • AmazonEKSVPCResourceController

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "cloudwatch:PutMetricData"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "ec2:DescribeAccountAttributes",
               "ec2:DescribeAddresses",
               "ec2:DescribeInternetGateways"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
Copy

IAM Role: {key}-addon-vpc-cni-Role

AWS-managed policies:

  • AmazonEKS_CNI_Policy

IAM Role: {key}-eks-role

AWS-managed policies:

  • AmazonEBSCSIDriverPolicy

  • AmazonEC2ContainerRegistryReadOnly

  • AmazonEKS_CNI_Policy

  • AmazonEKSWorkerNodePolicy

  • AmazonSSMManagedInstanceCore

  • AutoScalingFullAccess

  • ElasticLoadBalancingFullAccess

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "ec2:CreateSecurityGroup",
               "ec2:CreateTags"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:ec2:{Region}:{Account_ID}:security-group/*",
               "arn:aws:ec2:{Region}:{Account_ID}:vpc/{VPC_ID}"
            ],
            "Sid": "CreateOpenflowEKSSecurityGroupAndTags"
      },
      {
            "Action": [
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:DeleteSecurityGroup"
            ],
            "Condition": {
               "StringLike": {
                  "aws:ResourceTag/Name": "eks-cluster-sg-{key}-*"
               }
            },
            "Effect": "Allow",
            "Resource": [
               "arn:aws:ec2:{Region}:{Account_ID}:security-group/*"
            ],
            "Sid": "OpenflowManageEKSSecurityGroup"
      }
   ]
}
Copy

Note

{VPC_ID} represents the identifier of the VPC that was either created by BYOC or used by BYO-VPC.

IAM Role: oidc-service-role-{key}

{
   "Statement": [
      {
            "Action": [
               "secretsmanager:GetSecretValue",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:oidc-{key}*"
            ]
      }
   ],
   "Version": "2012-10-17"
}
Copy

IAM Role: dps-postgres-role-{key}

{
   "Statement": [
      {
            "Action": [
               "secretsmanager:GetSecretValue",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:postgres_creds-{key}*"
            ]
      }
   ],
   "Version": "2012-10-17"
}
Copy

IAM Role: dps-temporal-role-{key}

{
   "Statement": [
      {
            "Action": [
               "secretsmanager:GetSecretValue",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:temporal_creds-{key}*"
            ]
      }
   ],
   "Version": "2012-10-17"
}
Copy

IAM Role: dp-service-role-{key}

{
   "Statement": [
      {
            "Action": [
               "secretsmanager:GetSecretValue",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:dps_creds-{key}*",
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:snowflake-oauth2-{key}*"
            ]
      }
   ],
   "Version": "2012-10-17"
}
Copy

IAM Role: oauth2-role-{key}

{
   "Statement": [
      {
            "Action": [
               "secretsmanager:GetSecretValue",
               "secretsmanager:DescribeSecret",
               "secretsmanager:GetResourcePolicy",
               "secretsmanager:ListSecretVersionIds"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:secretsmanager:{Region}:{Account_ID}:secret:snowflake-oauth2-{key}*"
            ]
      }
   ],
   "Version": "2012-10-17"
}
Copy

IAM Role: {key}-nodegroup-NodeInstanceRole

AWS-managed policies:

  • AmazonEBSCSIDriverPolicy

  • AmazonEC2ContainerRegistryReadOnly

  • AmazonEKS_CNI_Policy

  • AmazonEKSWorkerNodePolicy

  • AmazonSSMManagedInstanceCore

  • AutoScalingFullAccess

  • ElasticLoadBalancingFullAccess

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "servicediscovery:CreateService",
               "servicediscovery:DeleteService",
               "servicediscovery:GetService",
               "servicediscovery:GetInstance",
               "servicediscovery:RegisterInstance",
               "servicediscovery:DeregisterInstance",
               "servicediscovery:ListInstances",
               "servicediscovery:ListNamespaces",
               "servicediscovery:ListServices",
               "servicediscovery:GetInstancesHealthStatus",
               "servicediscovery:UpdateInstanceCustomHealthStatus",
               "servicediscovery:GetOperation",
               "route53:GetHealthCheck",
               "route53:CreateHealthCheck",
               "route53:UpdateHealthCheck",
               "route53:ChangeResourceRecordSets",
               "route53:DeleteHealthCheck",
               "appmesh:*"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "autoscaling:DescribeAutoScalingGroups",
               "autoscaling:DescribeAutoScalingInstances",
               "autoscaling:DescribeLaunchConfigurations",
               "autoscaling:DescribeScalingActivities",
               "autoscaling:DescribeTags",
               "ec2:DescribeInstanceTypes",
               "ec2:DescribeLaunchTemplateVersions"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "autoscaling:SetDesiredCapacity",
               "autoscaling:TerminateInstanceInAutoScalingGroup",
               "ec2:DescribeImages",
               "ec2:GetInstanceTypesFromInstanceRequirements",
               "eks:DescribeNodegroup"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "iam:CreateServiceLinkedRole"
            ],
            "Condition": {
               "StringEquals": {
                  "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DescribeAccountAttributes",
               "ec2:DescribeAddresses",
               "ec2:DescribeAvailabilityZones",
               "ec2:DescribeInternetGateways",
               "ec2:DescribeVpcs",
               "ec2:DescribeVpcPeeringConnections",
               "ec2:DescribeSubnets",
               "ec2:DescribeSecurityGroups",
               "ec2:DescribeInstances",
               "ec2:DescribeNetworkInterfaces",
               "ec2:DescribeTags",
               "ec2:GetCoipPoolUsage",
               "ec2:DescribeCoipPools",
               "elasticloadbalancing:DescribeLoadBalancers",
               "elasticloadbalancing:DescribeLoadBalancerAttributes",
               "elasticloadbalancing:DescribeListeners",
               "elasticloadbalancing:DescribeListenerCertificates",
               "elasticloadbalancing:DescribeSSLPolicies",
               "elasticloadbalancing:DescribeRules",
               "elasticloadbalancing:DescribeTargetGroups",
               "elasticloadbalancing:DescribeTargetGroupAttributes",
               "elasticloadbalancing:DescribeTargetHealth",
               "elasticloadbalancing:DescribeTags"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "cognito-idp:DescribeUserPoolClient",
               "acm:ListCertificates",
               "acm:DescribeCertificate",
               "iam:ListServerCertificates",
               "iam:GetServerCertificate",
               "waf-regional:GetWebACL",
               "waf-regional:GetWebACLForResource",
               "waf-regional:AssociateWebACL",
               "waf-regional:DisassociateWebACL",
               "wafv2:GetWebACL",
               "wafv2:GetWebACLForResource",
               "wafv2:AssociateWebACL",
               "wafv2:DisassociateWebACL",
               "shield:GetSubscriptionState",
               "shield:DescribeProtection",
               "shield:CreateProtection",
               "shield:DeleteProtection"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:RevokeSecurityGroupIngress"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:CreateSecurityGroup"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:CreateTags"
            ],
            "Condition": {
               "Null": {
                  "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
               },
               "StringEquals": {
                  "ec2:CreateAction": "CreateSecurityGroup"
               }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:*:*:security-group/*"
      },
      {
            "Action": [
               "ec2:CreateTags",
               "ec2:DeleteTags"
            ],
            "Condition": {
               "Null": {
                  "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                  "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
               }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:*:*:security-group/*"
      },
      {
            "Action": [
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:RevokeSecurityGroupIngress",
               "ec2:DeleteSecurityGroup"
            ],
            "Condition": {
               "Null": {
                  "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "elasticloadbalancing:CreateLoadBalancer",
               "elasticloadbalancing:CreateTargetGroup"
            ],
            "Condition": {
               "Null": {
                  "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "elasticloadbalancing:CreateListener",
               "elasticloadbalancing:DeleteListener",
               "elasticloadbalancing:CreateRule",
               "elasticloadbalancing:DeleteRule"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "elasticloadbalancing:AddTags",
               "elasticloadbalancing:RemoveTags"
            ],
            "Condition": {
               "Null": {
                  "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                  "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
               }
            },
            "Effect": "Allow",
            "Resource": [
               "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
               "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
               "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ]
      },
      {
            "Action": [
               "elasticloadbalancing:AddTags",
               "elasticloadbalancing:RemoveTags"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
               "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
               "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
               "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
            ]
      },
      {
            "Action": [
               "elasticloadbalancing:ModifyLoadBalancerAttributes",
               "elasticloadbalancing:SetIpAddressType",
               "elasticloadbalancing:SetSecurityGroups",
               "elasticloadbalancing:SetSubnets",
               "elasticloadbalancing:DeleteLoadBalancer",
               "elasticloadbalancing:ModifyTargetGroup",
               "elasticloadbalancing:ModifyTargetGroupAttributes",
               "elasticloadbalancing:DeleteTargetGroup"
            ],
            "Condition": {
               "Null": {
                  "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "elasticloadbalancing:AddTags"
            ],
            "Condition": {
               "Null": {
                  "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
               },
               "StringEquals": {
                  "elasticloadbalancing:CreateAction": [
                        "CreateTargetGroup",
                        "CreateLoadBalancer"
                  ]
               }
            },
            "Effect": "Allow",
            "Resource": [
               "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
               "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
               "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ]
      },
      {
            "Action": [
               "elasticloadbalancing:RegisterTargets",
               "elasticloadbalancing:DeregisterTargets"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
      },
      {
            "Action": [
               "elasticloadbalancing:SetWebAcl",
               "elasticloadbalancing:ModifyListener",
               "elasticloadbalancing:AddListenerCertificates",
               "elasticloadbalancing:RemoveListenerCertificates",
               "elasticloadbalancing:ModifyRule"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "route53:ChangeResourceRecordSets"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:route53:::hostedzone/*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "route53:GetChange"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:route53:::change/*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "route53:ListResourceRecordSets",
               "route53:ListHostedZonesByName"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "ec2:CreateSnapshot",
               "ec2:AttachVolume",
               "ec2:DetachVolume",
               "ec2:ModifyVolume",
               "ec2:DescribeAvailabilityZones",
               "ec2:DescribeInstances",
               "ec2:DescribeSnapshots",
               "ec2:DescribeTags",
               "ec2:DescribeVolumes",
               "ec2:DescribeVolumesModifications"
            ],
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:CreateTags"
            ],
            "Condition": {
               "StringEquals": {
                  "ec2:CreateAction": [
                        "CreateVolume",
                        "CreateSnapshot"
                  ]
               }
            },
            "Effect": "Allow",
            "Resource": [
               "arn:aws:ec2:*:*:volume/*",
               "arn:aws:ec2:*:*:snapshot/*"
            ]
      },
      {
            "Action": [
               "ec2:DeleteTags"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:ec2:*:*:volume/*",
               "arn:aws:ec2:*:*:snapshot/*"
            ]
      },
      {
            "Action": [
               "ec2:CreateVolume"
            ],
            "Condition": {
               "StringLike": {
                  "aws:RequestTag/ebs.csi.aws.com/cluster": "true"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:CreateVolume"
            ],
            "Condition": {
               "StringLike": {
                  "aws:RequestTag/CSIVolumeName": "*"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DeleteVolume"
            ],
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/ebs.csi.aws.com/cluster": "true"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DeleteVolume"
            ],
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/CSIVolumeName": "*"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DeleteVolume"
            ],
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/kubernetes.io/created-for/pvc/name": "*"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DeleteSnapshot"
            ],
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/CSIVolumeSnapshotName": "*"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      },
      {
            "Action": [
               "ec2:DeleteSnapshot"
            ],
            "Condition": {
               "StringLike": {
                  "ec2:ResourceTag/ebs.csi.aws.com/cluster": "true"
               }
            },
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "route53:ChangeResourceRecordSets"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:route53:::hostedzone/*"
      }
   ]
}
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "route53:ListHostedZones",
               "route53:ListResourceRecordSets",
               "route53:ListTagsForResource"
            ],
            "Effect": "Allow",
            "Resource": "*"
      }
   ]
}
Copy