Data Migration & Validation

Snowflake AIM Data Migration and Validation (AIM DMV) provides a fault-tolerant, scalable way to move data from external sources into Snowflake and verify that migrated data matches the source. Both features share the same Orchestrator and Workers, the same SNOWCONVERT_AI metadata database, and the same deployment options.

Use Data migration when you are moving data from a system you plan to decommission. Use Data validation when you need confidence that migrated data is correct before cutover.

Architecture overview

AIM DMV uses two main components: an Orchestrator and one or more Workers.

  • The Orchestrator connects to your Snowflake account. It requires privileges to create and operate the SNOWCONVERT_AI database, where workflow, task, and result metadata is stored.
  • One or more Workers connect to both the source system and Snowflake. For migration, Workers read data from the source, upload files to a Snowflake stage, and the Orchestrator loads them with COPY INTO. For validation, Workers run comparison queries on both sides and write results that the Orchestrator evaluates.
  • Validation results are ingested into shared results tables via Snowpipe by default.

Each Worker that executes validation tasks must have the validation runtime available. Workers that only run data-migration tasks won’t pick up validation tasks from the queue.

Deployment options

The Orchestrator and Workers can be deployed in multiple ways:

  • Both on Snowpark Container Services (in the Snowflake account).
  • Both in your environment, including custom hardware, virtual machines, or containers.
  • Orchestrator on Snowpark Container Services and Workers in your environment, or the other way around.

See Deploying on Snowpark Container Services for compute pools, image repositories, and network access setup.

Prerequisites

Before you use AIM DMV, make sure the following are in place:

  • Snowflake AIM Agent for Data Warehouses: The agent installs and manages the Orchestrator and Worker runtimes and their dependencies as needed.
  • Snowflake access: Connections for the Orchestrator and Workers in your Snowflake config.toml or connections.toml, using a role that can create the SNOWCONVERT_AI database and its objects.
  • Source connectivity: Platform-specific drivers on Workers. See the per-platform pages under Data migration and Data validation.
  • Hybrid Tables: On bootstrap, AIM DMV probes whether Hybrid Tables are enabled and available in your Snowflake account and region. When they are, SNOWCONVERT_AI metadata objects (for example the task queue) are created as Hybrid Tables. When they are not, AIM DMV falls back to standard tables instead. Workflows still run in both cases, but metadata operations are slower without Hybrid Tables. Review Hybrid tables and Hybrid tables limitations.
  • Snowpark Container Services (optional): Required only when you deploy the Orchestrator or Workers on Snowflake compute. See the Snowpark Container Services overview.

Connecting to Snowflake with a PAT

Use a programmatic authentication method for Orchestrator and Worker Snowflake connections. Programmatic Access Tokens (PATs) and key-pair authentication are both suitable. Interactive methods such as browser-based SSO are not recommended: AIM DMV opens and closes many Snowflake connections during a workflow, and interactive auth flows are a poor fit for that pattern.

If you use PATs, you’ll need to establish a Network Policy or temporarily bypass the requirement from Snowsight. See Programmatic access tokens.

Managing Workers

The number of Workers (and threads per Worker) has the greatest impact on completion time. Consider the following:

  • Don’t run two Workers on the same machine; increase thread count instead.
  • Network bandwidth is shared between threads of a Worker.
  • Keep a low Worker count to avoid overloading your source system.
  • Consider stopping Workers when the source system is under heavy load.

Monitoring workflows

Workflow and validation metadata is stored in the SNOWCONVERT_AI database. Filter almost every query by WORKFLOW_ID to scope results to a single run. See The SNOWCONVERT_AI database for tables, views, workflow management procedures, and sample queries.

Related content