Deploying on Amazon ECS¶
Run the Worker and the Orchestrator on Amazon ECS when you want Workers close to your source system without operating a Kubernetes cluster. This page runs both as ECS services on AWS Fargate. For a Kubernetes cluster, see Deploying on Amazon EKS.
The AIM DMV images are published to the Snowflake image registry, which requires a Snowflake login. Copy the images to Amazon ECR and run them from there. Keep source and Snowflake credentials in AWS Secrets Manager, and reference them from the task definitions with aws-sm:/// references.
Before you start¶
- An Amazon ECS cluster. This page creates one named
dmv. - An Amazon ECR repository for each image:
data-exchange-workeranddata-migration-orchestrator. - A workstation or CI runner with Docker, the Snowflake CLI, the AWS CLI, and the Session Manager plugin for the AWS CLI. Run the copy steps and the health checks there.
- Access to the image repository
snowflake.images.snowflake_imagesin your account. - Source and Snowflake credentials stored in AWS Secrets Manager. See AWS Secrets Manager.
- VPC subnets for the tasks, with outbound access to the destinations listed in Networking.
1. Copy the images to Amazon ECR¶
The images are built for linux/amd64. On Fargate, set the task’s CPU architecture to X86_64. Don’t use ARM64.
List the available versions of both images with the Snowflake CLI. The repository contains other images, so the command filters for the two AIM DMV images:
Pick the newest version that both data-exchange-worker and data-migration-orchestrator list. Use that version number for both images. Don’t use the tag latest.
Run the following steps from a workstation or CI job. Replace <version> with the version you chose, and replace the other placeholders with your Snowflake connection name, AWS account ID, and region:
Repeat these steps when you upgrade AIM DMV. Run the copy from a workstation or CI job, rather than storing the Snowflake registry login in AWS.
2. Create IAM roles¶
ECS uses two kinds of roles. The task execution role lets ECS pull the image from ECR and write logs to Amazon CloudWatch Logs. The task role is the identity the container uses to call AWS, so it is the role that reads Secrets Manager. Create one execution role, and one task role each for the Worker and the Orchestrator.
All three roles trust the ECS tasks service. Save this trust policy as trust.json:
Create the roles, and attach the AWS managed execution policy to the execution role:
Grant each task role secretsmanager:GetSecretValue only on the secrets it reads. The Worker reads the source and Snowflake secrets. The Orchestrator reads only the Snowflake secret. The ssmmessages actions let you run the health checks in step 5 with ECS Exec.
Worker task role policy, saved as worker-policy.json:
Orchestrator task role policy, saved as orchestrator-policy.json:
Secret ARNs end with a random suffix, so the -* at the end of each resource matches it. If the secret uses a customer-managed KMS key, also allow kms:Decrypt on that key.
Don’t set access keys in the task definition. The AWS SDK in the container picks up the task role.
3. Create the cluster and networking¶
Create the cluster, a CloudWatch Logs log group, and a security group for the tasks. The tasks only make outbound connections, so the security group needs no inbound rules:
Networking¶
Fargate tasks use the awsvpc network mode, so each task gets its own network interface in the subnets you choose. Use private subnets with a NAT gateway, and set assignPublicIp=DISABLED when you create the services. The tasks need outbound access to:
- Your source system, for example Amazon Redshift on port
5439. - Snowflake.
- Amazon ECR and Amazon S3, to pull the images.
secretsmanager.<region>.amazonaws.com, to resolveaws-sm:///references.- Amazon CloudWatch Logs, for the
awslogslog driver. ssmmessages.<region>.amazonaws.com, for ECS Exec.- The Debian package mirrors (
deb.debian.org). The Worker image installs the source driver from them at start.
If your account blocks egress to unknown IP addresses, allow these destinations from the NAT gateway’s address.
4. Register the task definitions and start the services¶
Worker¶
Put source and Snowflake settings on the container as environment variables, the same way SPCS does. The values can be aws-sm:/// references. They aren’t credentials, so the task definition can live in your repository.
Don’t set command or entryPoint. The image’s entrypoint installs the source driver, writes the Worker configuration from these variables, and starts the agent. Set AWS_DEFAULT_REGION to the region that holds your secrets. The AWS SDK doesn’t read AWS_REGION.
Save this task definition as worker-taskdef.json. This example authenticates to Snowflake with a key pair:
DATA_SOURCE_HOST,DATA_SOURCE_PORT, andDATA_SOURCE_DATABASEcan beaws-sm:///references, as shown, or plain values. SetDATA_SOURCE_TYPEto your source database, for exampleredshift.SNOWFLAKE_CONNECTION_NAMEmust beMY_SNOWFLAKE_CONNECTION, which is the name the image writes into the Snowflake connections file. The default@SPCS_CONNECTIONonly works on SPCS.SNOWFLAKE_PRIVATE_KEYmakes the image write the key under$HOME/.snowflake/with mode0600, and a named connection that usesauthenticator = SNOWFLAKE_JWT. Storeaccount,user, and the PEM in theprivate_keyfield of theaim-dmv/prod/snowflakesecret. Therole,warehouse,database, andschemafields are read if you set the matching environment variables.- For password authentication, replace
SNOWFLAKE_PRIVATE_KEYwithSNOWFLAKE_PASSWORDset toaws-sm:///aim-dmv/prod/snowflake#password. Don’t set both. AGENT_AFFINITYroutes workflows to this Worker. The Orchestrator uses the same value. See Affinity and Environment variables.- The
doctorcommand checks the Worker’s resources. With 4 vCPU, 8 GB of memory, and 60 GiB of ephemeral storage, all its checks pass. On a smaller task, such as 1 vCPU and 2 GB with the default 20 GiB of storage, the memory and disk checks fail. initProcessEnabledis recommended for ECS Exec sessions.
Orchestrator¶
The Orchestrator image starts its loop by default, so don’t set command. Save this task definition as orchestrator-taskdef.json:
The Orchestrator’s DATA_MIGRATION_ORCHESTRATOR_AFFINITY must match the Worker’s AGENT_AFFINITY. For password authentication, replace SNOWFLAKE_PRIVATE_KEY with SNOWFLAKE_PASSWORD, as for the Worker. For the environment variables each component reads, see Orchestrator and Worker settings.
Start the services¶
Register both task definitions, then run each one as a service with one task. --enable-execute-command turns on ECS Exec for the health checks:
To change a task definition later, register it again and point the service at the new revision with aws ecs update-service --cluster dmv --service dmv-worker --task-definition dmv-worker.
5. Check the deployment¶
Run the health checks in each task with ECS Exec. Both commands print the results as JSON. Check that every check passes. Pass -c so the Worker’s doctor reads the configuration the entrypoint wrote. The entrypoint prints that path at start, in the Agent configuration generated at line of the Worker’s log:
The Orchestrator’s doctor checks Snowflake connectivity. Its workflow configuration checks are skipped unless you pass a configuration path.
Check the Orchestrator’s log in CloudWatch Logs. Once it’s polling, it logs PULL_TASKS returned 0 row(s) about every 20 seconds until you create workflows:
Once both services are running and the checks pass, create your workflows. The Orchestrator picks them up and assigns the tasks to Workers with a matching affinity.
Troubleshooting¶
| Symptom | Likely cause | What to do |
|---|---|---|
Role is not valid from register-task-definition | A role ARN in the task definition is wrong. | Check executionRoleArn and taskRoleArn against aws iam get-role. |
CannotPullContainerError | The image isn’t in ECR at that tag, or the execution role can’t read ECR. | Confirm the tag exists with aws ecr describe-images. Attach AmazonECSTaskExecutionRolePolicy to the execution role. |
exec format error | The task runs on ARM64. | Set runtimePlatform.cpuArchitecture to X86_64. |
Access denied when resolving an aws-sm:/// reference | The task role can’t read that secret. The execution role’s permissions don’t apply to the container. | Check the task role’s policy and the secret ARN. |
| Secret not found, or a region error | AWS_DEFAULT_REGION is missing or points to a region without the secret. | Set AWS_DEFAULT_REGION. AWS_REGION is not read. Replicate the secret to that region if needed. |
SessionManagerPlugin is not found | The Session Manager plugin isn’t installed where you run aws ecs execute-command. | Install the Session Manager plugin. |
execute-command fails to start a session | The service was created without --enable-execute-command, or the task role lacks the ssmmessages actions. | Update the service with --enable-execute-command and --force-new-deployment, and check the task role’s policy. |
The Worker’s doctor fails the memory or disk check | The task is too small. | Raise cpu, memory, and ephemeralStorage in the task definition. |
| Connection timeout to the source or to Snowflake | The task’s subnet or security group doesn’t allow the destination. | Check the route from the task subnets to the source and to Snowflake. See Networking. |
After you rotate a secret, restart the tasks with aws ecs update-service --cluster dmv --service <service-name> --force-new-deployment. Resolved secrets are cached for SECRET_MANAGERS_CACHE_TTL_SECONDS, which defaults to 300 seconds. See Supported vaults for the full list of AWS Secrets Manager limitations.