SnowConvert AI: Data migration [Preview]¶

Data Migration uses the data-migration skill from the migration-plugin. For more details, contact snowconvert-support@snowflake.com.

The Data Migration feature of SnowConvert provides a fault-tolerant, scalable solution for moving data from external sources into Snowflake. This feature is specifically designed for cases where you are moving data from a system you plan to decommission. For replication purposes, other solutions are available that might better fit your use case.

Architecture overview¶

The Data Migration feature uses two main components: an Orchestrator and one or more Workers.

• The Orchestrator connects to the Snowflake account. It requires privileges to create and operate the SNOWCONVERT_AI database, where metadata is stored.
• One or more Workers connect to both the source system and the Snowflake account. Workers read data from the source system and upload it to a Snowflake stage. Workers pick up tasks created by the Orchestrator and process them in parallel.
• Files uploaded to the Snowflake stage are copied into the target tables using a COPY INTO statement submitted and monitored by the Orchestrator.

Deployment options¶

The Orchestrator and Workers can be deployed in multiple ways:

• Both on Snowpark Container Services (in the Snowflake account).
• Both in the customer’s environment, including custom hardware, virtual machines, or containers.
• Orchestrator on Snowpark Container Services and Workers in the customer’s environment, or the other way around.

The following requirements apply to the environment where the Orchestrator and Workers run:

• Use the Snowflake Migration Agent with Cortex Code, or the SnowConvert AI CLI (scai) from an initialized SnowConvert AI project. The skill and CLI install and manage the worker and orchestrator runtimes and their dependencies when needed.
• Workers typically require an ODBC driver to connect to the source system (unless the source supports a pure-Python driver, such as Teradata with teradatasql).
• The Orchestrator must be able to connect to the Snowflake account using a role that has privileges to create the SNOWCONVERT_AI database and create schemas and objects within it.

Prerequisites¶

Before you use Data Migration, make sure the following are in place:

• SnowConvert AI project and skill: Cortex Code with the Snowflake Migration Agent enabled, or the SnowConvert AI CLI (scai) installed and a migration project on disk. The skill installs scai, connection tooling, and worker dependencies on first use—see the skill documentation for prerequisites.
• Snowflake access: Connections for the Orchestrator and Workers in your Snowflake config.toml or connections.toml, using a role that can create the SNOWCONVERT_AI database and its objects. The first time the Orchestrator starts, it creates that database and related resources if they do not exist yet. On later runs, use a role that can administer SNOWCONVERT_AI and its objects; sticking with the same role you used for the initial creation is the simplest way to avoid permission issues.
• Source connectivity: For typical source databases, an ODBC driver on the machine where Workers run. Programmatic Access Tokens (PATs) are recommended for Snowflake connections; see Connecting to Snowflake with a PAT.
• Hybrid Tables: Hybrid Tables must be enabled and available in your Snowflake account and region for this feature. Review Hybrid tables and Hybrid tables limitations so you understand relevant platform requirements.
• Snowpark Container Services (optional): If you deploy the Orchestrator or Workers on Snowflake compute, your account needs SPCS. See the Snowpark Container Services overview. Running both components outside Snowflake does not require SPCS.

Source-platform specifics¶

The Worker connects to your source using the TOML [connections.source.*] sections described in Worker configuration. Use the tabs below for connectivity, extraction style, and target options that are unique to each platform.

For Teradata only, the migration workflow’s extraction.strategy chooses among regular (default: partitioned SQL through the Worker to Parquet, then the internal migration stage), write_nos (Teradata WRITE_NOS writes Parquet to S3, Azure Blob, or GCS; Snowflake loads via an external stage you align to that bucket), and tpt (the Worker runs TPT EXPORT with tbuild, converts delimited output to Parquet, then uses the same internal stage path as regular).

Setup¶

Installation¶

Use the Snowflake Migration Agent in Cortex Code (recommended) so the agent can drive setup, or install the SnowConvert AI CLI (scai) and work from a SnowConvert AI project directory. The skill registers the migration plugin and ensures scai and dependencies are available.

• Installing the CLI (via the skill or your platform installer) does not require Snowpark Container Services (SPCS). SPCS is required only when you deploy the Orchestrator (or Workers) on Snowflake compute using container services.

Usage¶

To migrate data using this solution, complete the following high-level steps:

1. Start the Orchestrator.
2. Start the Workers.
3. Create a Data Migration Workflow.
4. Monitor the Data Migration Workflow until completion.

A Data Migration Workflow is essentially an action or goal for the system to complete, such as migrating a specific set of tables with a given configuration. You can submit multiple workflows simultaneously and monitor them. The Orchestrator breaks Data Migration Workflows into smaller tasks, which typically involves splitting a table into partitions before extracting its data and loading it to Snowflake.

You drive orchestration, workers, and workflows through the scai data … commands below. In Cortex Code, the data-migration skill wraps the same flows: ask the agent to run cloud data migration steps, or follow the Snowflake Migration Agent guide for guided prompts.

Using SCAI CLI¶

Cloud data commands are grouped under scai data using nested verbs (migrate, worker, orchestrator, …). The following deprecated top-level names remain available and print a short warning pointing to the new path; prefer the nested commands for new scripts and documentation.

Global options such as --json (structured stdout for automation), --log-debug, --no-auto-update, and connection overrides (-c / --connection, --warehouse, --role) apply wherever the CLI exposes them; run scai data <branch> <command> --help for the exact surface.

Starting the Orchestrator on SPCS¶

Use this command to create or resume the Orchestrator service on Snowpark Container Services. Stop (or suspend) the service when it is no longer needed; the CLI reminds you to run scai data orchestrator stop after workflows that start the service.

# Default Snowflake connection from your environment / project:
scai data orchestrator setup --compute-pool MY_COMPUTE_POOL

# Named Snowflake connection:
scai data orchestrator setup --compute-pool MY_COMPUTE_POOL --connection MY_SNOWFLAKE_CONNECTION

# Optional custom container image (advanced):
# scai data orchestrator setup --compute-pool MY_COMPUTE_POOL --custom-image /DB/SCHEMA/REPO/IMAGE:tag

scai data orchestrator setup --help

# Suspend the service (default):
scai data orchestrator stop

# Drop the service instead of suspending:
scai data orchestrator stop --drop

scai data orchestrator stop --help

Starting the Workers¶

Start a Worker with a TOML configuration file. See Worker configuration. You can emit a starter template (with placeholders or pre-filled source details when a project is present) using --auto-config.

# Start with a configuration file (positional argument):
scai data worker start my-worker-config.toml

# Same, with an explicit Snowflake connection profile:
scai data worker start my-worker-config.toml --connection my-snowflake

# Generate a template only (default path ~/.snowflake/scai/dew_configuration.toml, or pass a path):
scai data worker start --auto-config
scai data worker start --auto-config /path/to/my-worker-config.toml

scai data worker start --help

scai data worker start installs and runs the Data Exchange Worker in the foreground (the worker process managed by the CLI). Foreground worker start does not support --json; use --auto-config with --json if you need JSON output for template generation only.

Generating a workflow configuration file¶

From an SCAI project directory, generate a YAML workflow configuration (default file name data-migration-config.yaml in the current directory). You can author YAML by hand as long as it matches the workflow object model described in Workflow configuration reference.

scai data migrate generate-config

scai data migrate generate-config --where "source.schema = 'public'"

scai data migrate generate-config -o my-config.yaml

# Optional workflow affinity (must match Workers that should pick up the workflow):
scai data migrate generate-config --affinity my-team

scai data migrate generate-config --help

Creating a Data Migration Workflow (and optional local Orchestrator / Worker)¶

scai data migrate create-workflow submits a workflow to Snowflake. Options combine Snowflake connection overrides, optional SPCS --start-service, and optional local Orchestrator / Worker processes.

# Create workflow and return immediately:
scai data migrate create-workflow --config my-data-migration-config.yaml --connection my-snowflake

# Create and wait until completion:
scai data migrate create-workflow --config my-data-migration-config.yaml --connection my-snowflake --watch

# Start SPCS service, then create workflow:
scai data migrate create-workflow --config my-data-migration-config.yaml --start-service \
  -p MY_COMPUTE_POOL --connection my-snowflake

scai data migrate create-workflow --help

Local “all-in-one” run from a project (migrate start)¶

scai data migrate start is a narrow wrapper intended for local runs from an SCAI project: if the config file is missing and you did not pass --no-auto-config, it generates a default workflow config, then delegates to create-workflow with --start-worker, --start-orchestrator, and --watch forced on (no --start-service / compute pool on this path). Use --no-pip-upgrade to skip upgrading bundled worker/orchestrator dependencies when the CLI prepares the local environment.

scai data migrate start
scai data migrate start --config my-data-migration-config.yaml --connection my-snowflake
scai data migrate start --help

Monitoring and listing workflows¶

# One-shot status:
scai data migrate status DATA_MIGRATION_WORKFLOW_xx_yy_zz

# Watch until completion:
scai data migrate status DATA_MIGRATION_WORKFLOW_xx_yy_zz --watch

scai data migrate status --help

# Recent workflows (default limit 20):
scai data migrate list

scai data migrate list --status running --connection my-snowflake
scai data migrate list --limit 50

scai data migrate list --help

The same observability features in Snowflake—including the DATA_MIGRATION_DASHBOARD Streamlit dashboard—apply when you use scai or the migration skill.

Workflow lifecycle and Snowflake observability¶

Each workflow goes through different stages throughout its lifecycle:

• Pending: No tasks have been created for this workflow yet.
• Executing: Tasks have been created for this workflow and there are still tasks that have not reached a terminal state (COMPLETED or FAILED).
• Completed: All tasks have reached a terminal state (COMPLETED or FAILED).

In the SNOWCONVERT_AI.DATA_MIGRATION schema, the following tables and views can be queried to understand the status of one or more workflows:

• WORKFLOW: Contains one row per workflow, including start/end time, status, and configuration.
• TABLE_PROGRESS_WITH_EXAMPLE_ERROR: Contains one row per table being migrated as part of a workflow. Includes information about how many partitions are in each stage (extraction, loading, completed, or failed), as well as related errors. Can be filtered by WORKFLOW_ID.
• DATA_MIGRATION_ERROR: For each partition of a table being migrated, contains the first known error that affected the migration of that partition. Can be filtered by WORKFLOW_ID.

In the same schema, the DATA_MIGRATION_DASHBOARD Streamlit dashboard can be used to monitor workflows. This dashboard presents data from those tables and views.

Workflow configuration for scai data migrate create-workflow uses YAML (.yaml / .yml) aligned with your project; supported source platforms in that configuration include SQL Server, Amazon Redshift, Oracle, and Teradata. The full object model is documented in Workflow configuration reference.

Redshift UNLOAD¶

For Redshift, it is recommended to use the unload extraction strategy. This works as follows:

• Large query results are written directly to an S3 bucket instead of being downloaded to the machine running the Worker.
• On the Snowflake side, an external stage is configured to reference the corresponding S3 bucket, so that COPY INTO statements can be executed directly from that stage.

For configuration details, see ExtractionStrategy model.

Incremental synchronization¶

You can migrate tables and then re-migrate them in the future, moving only the data that has changed. For more details, see SynchronizationStrategy model.

Considerations and recommendations¶

Connecting to Snowflake with a PAT¶

It is recommended to use Programmatic Access Tokens (PAT) for connections used by the Orchestrator and Workers. This ensures there is no need to constantly authenticate through the browser or with an authenticator app. You will need to establish a Network Policy or temporarily bypass the requirement for a Network Policy (this can be done from Snowsight).

Running Orchestrator and Workers on SPCS¶

To leverage Snowflake compute for these tasks:

1. Prepare Docker images that bundle the Orchestrator and Worker runtimes and configuration expected by your SnowConvert AI release (see Snowflake guidance for SPCS service images).
2. Push those Docker images to an Image Repository in Snowflake.
3. Execute the Orchestrator and/or Worker images using Snowpark Container Services.

Keep the following in mind:

• It is recommended to execute them as Services, not Jobs.
• It is possible to run only one component (Orchestrator or Workers) in SPCS and the other on a different platform.
• It is a good practice to monitor the SPCS service and suspend it when it is not being used.
• Depending on the network configuration of the source system, you may need to configure an External Access Integration so that these services can connect to your source system.

Initial testing¶

It is recommended to deploy the DDL for the tables you want to migrate before starting data migration. This ensures the target type matches the behavior you want to see in the table and its related views and procedures. You can convert DDL from your source dialect into Snowflake SQL using the code conversion capabilities of SnowConvert AI and/or Cortex Code.

For an early test run, use a separate workflow configuration whose tables array lists only the table or small set of tables you want to validate. On each of those entries, set whereClauseCriteria to an SQL-like predicate (as you would in a WHERE clause) so only a subset of rows is migrated, for example a bounded primary-key range or a narrow date range in the source dialect. You can also set a small partitionSize (for example maxRowsPerPartition) to keep partitions tiny during the test. After you confirm connectivity, performance, and results, create your full workflow: remove or relax whereClauseCriteria and use "auto" or your production partitionSize settings.

Managing Workers¶

The time it takes to complete a workflow depends on many variables, but the number of Workers (and threads per Worker) has the greatest impact, as it determines how many extraction tasks can be executed in parallel. Consider the following:

• It is not necessary to run two Workers on the same machine. If you want more parallelism on a single machine, increase the thread count instead.
• Network bandwidth greatly affects Worker speed and is shared between threads of a Worker.
• Even with many Workers and threads running in parallel, the source system might not have enough resources to handle the load.
• Keep a low Worker count to avoid overloading your source system.
• Consider stopping some or all Workers when the source system is already under heavy load from other operations.

Configuration reference¶

Workflow configuration reference¶

The Data Migration Workflow configuration file is a JSON object. The following sections describe its structure and properties.

Top-level object¶

When defaultTableConfiguration is present, each object in tables is merged with those defaults: shared fields apply to every table unless the same field is set again on a specific table entry, in which case the table-level value wins.

TableConfiguration model¶

Defines the settings for migrating a single table.

SourceTargetIdentifier model¶

A nested object used within TableConfiguration to specify a database object. For source, use only the properties in the following table. For target, you can also set the optional properties in Additional target properties.

Additional target properties¶

The following optional fields apply only to the target object (not to source).

Iceberg configuration (target.icebergConfig)¶

Used when target.tableType is "iceberg". Account setup (external volumes, catalog integrations, stages, and privileges) follows Snowflake’s Iceberg documentation; see Apache Iceberg™ tables, Create an Iceberg table, and Configure an external volume.

Partition size (partitionSize)¶

Controls how large each partition should be during extraction. You can use a string or an object.

When you use the object form, specify only one of targetSizeMb or maxRowsPerPartition.

auto (default):

"partitionSize": "auto"

Target size in MB:

"partitionSize": { "targetSizeMb": 2048 }

Maximum rows per partition:

"partitionSize": { "maxRowsPerPartition": 500000 }

ColumnTypeMapping model¶

A nested object used within TableConfiguration to specify type mappings for a column.

ColumnNameMapping model¶

A nested object used within TableConfiguration to specify column name mappings.

ExtractionStrategy model¶

Configures the method for data extraction.

Extraction: regular (default)

"extraction": {
  "strategy": "regular"
}

Extraction: unload (Redshift only)

"extraction": {
  "strategy": "unload",
  "externalStage": "MY_DB.MY_SCHEMA.S3_EXTERNAL_STAGE"
}

Extraction: WRITE_NOS (Teradata only)

"extraction": {
  "strategy": "write_nos",
  "externalStage": "MY_DB.MY_SCHEMA.TD_WRITE_NOS_STAGE"
}

Extraction: TPT (Teradata only)

"extraction": {
  "strategy": "tpt"
}

SynchronizationStrategy model¶

Configures the approach for incremental data syncing on subsequent runs.

Strategy: none (Default)

Performs a full extraction of all partition data on every run. No synchronization metadata is stored.

"synchronization": {
  "strategy": "none"
}

Use when data is small, changes are unpredictable, or guaranteed consistency is needed.

Strategy: checksum

Computes a hash of all column values for each partition on the source. Only changed partitions are cleared and re-extracted in the target.

"synchronization": {
  "strategy": "checksum"
}

Use when you need to detect any change in a partition but lack a reliable monotonic column (for example, dimension tables). Note that this requires a checksum computation on the source for every partition on every run.

Strategy: watermark

Tracks a monotonic column (timestamp, ID, or version) to sync only rows where the watermark value is greater than the maximum observed in the previous sync.

"synchronization": {
  "strategy": "watermark",
  "watermarkColumn": "UPDATED_AT"
}

Use when your table has a reliable monotonic column that increases on insert/update (for example, fact tables or event logs).

Example workflow: Redshift UNLOAD with Iceberg targets¶

The following workflow excerpt combines Redshift UNLOAD with Iceberg table targets, including Snowflake-managed Iceberg defaults and a per-table external catalog override:

{
  "defaultTableConfiguration": {
    "source": {
      "schemaName": "public",
      "databaseName": "analytics_db"
    },
    "target": {
      "schemaName": "public",
      "databaseName": "TARGET_DB",
      "tableType": "iceberg",
      "icebergConfig": {
        "catalog": "SNOWFLAKE",
        "externalVolume": "my_iceberg_ext_vol",
        "baseLocationPrefix": "migrations/redshift",
        "sourceDataStage": "@TARGET_DB.PUBLIC.ICEBERG_SOURCE_STAGE"
      }
    },
    "extraction": {
      "strategy": "unload",
      "externalStage": "TARGET_DB.PUBLIC.S3_EXTERNAL_STAGE"
    },
    "partitionSize": "auto"
  },
  "tables": [
    {
      "source": { "tableName": "customers" },
      "target": { "tableName": "customers" },
      "columnNamesToPartitionBy": ["customer_id"]
    },
    {
      "source": { "tableName": "events" },
      "target": {
        "tableName": "events",
        "tableType": "iceberg",
        "icebergConfig": {
          "catalog": "my_glue_catalog_integration",
          "externalVolume": "my_iceberg_ext_vol",
          "catalogTableName": "glue_db.events"
        }
      },
      "columnNamesToPartitionBy": ["event_id"]
    }
  ]
}

Affinity¶

By specifying an affinity for a Workflow, you are indicating that you want specific workers to help with the execution of that Workflow. This can be particularly useful in cases in which you want to have some workers extract from one source and other workers extract from a different source. The rules for matching workers with tasks are:

• A task without affinity will be picked up by any worker, independently of the worker’s affinity.
• A worker without affinity will pick up any task, independently of the task’s affinity.
• A task with a given affinity will not be picked up by a worker with different affinity.

Affinity only needs to be a String; its format is defined by the user. For example, all of these are valid: sql-server, DEV_SERVER, my_custom_server, ::blue::.

Worker configuration¶

This file configures the behavior and connections for the Data Exchange Worker. For cloud data migration and validation, set selected_task_source to "snowflake_stored_procedure" and provide a matching [task_source.snowflake_stored_procedure] section. That section’s connection_name is the Snowflake profile used to call the task-queue stored procedures (PULL_TASKS, COMPLETE_TASK, FAIL_TASK, and related objects). It is separate from [connections.target.snowflake_connection_name], which controls JDBC/ODBC sessions the Worker opens for data movement and validation queries—typically you set both to the same logical Snowflake account connection name.

An example configuration file for cloud migration:

selected_task_source = "snowflake_stored_procedure"

[task_source.snowflake_stored_procedure]
connection_name = "connection_name"

[application]
max_parallel_tasks = 4
task_fetch_interval = 30
# lease_refresh_interval = 120
# Optional: only if the Orchestrator uses CUSTOM_SNOWFLAKE_* overrides for metadata location
# snowflake_database_for_metadata = "SNOWCONVERT_AI"
# snowflake_schema_for_data_migration_metadata = "DATA_MIGRATION"

# SQL Server connection (standard authentication)
[connections.source.sqlserver]
username = "username"
password = "password"
database = "database_name"
host = "127.0.0.1"
port = 1433

# Amazon Redshift — define only the profile that matches your migration source (examples; keep one block):
# [connections.source.redshift]
# username = "demo-user"
# database = "snowconvert_demo"
# auth_method = "iam-provisioned-cluster"
# cluster_id = "migrations-aws"
# region = "us-west-2"
# access_key_id = "your-access-key-id"
# secret_access_key = "your-secret-access-key"
#
# [connections.source.redshift]
# username = "myuser"
# password = "mypassword"
# database = "mydatabase"
# host = "my-cluster.abcdef123456.us-west-2.redshift.amazonaws.com"
# port = 5439
# auth_method = "standard"

# Snowflake target connection (data sessions)
[connections.target.snowflake_connection_name]
connection_name = "connection_name"

Source connection configuration examples¶

The following examples show common source connection shapes. Table keys under [connections.source.<key>] are arbitrary identifiers; use names that match how you reference the source in your environment.

1. SQL Server (standard authentication)

[connections.source.sqlserver]
username = "username"
password = "password"
database = "database_name"
host = "127.0.0.1"
port = 1433

2. Amazon Redshift (IAM authentication)

[connections.source.redshift]
username = "demo-user"
database = "demo_db"
auth_method = "iam-provisioned-cluster"
cluster_id = "my-aws-cluster"
region = "us-west-2"
access_key_id = "your-access-key-id"
secret_access_key = "your-secret-access-key"
# Optional fields for UNLOAD strategy
# unload_s3_bucket = "my-migrations-bucket"
# unload_iam_role_arn = "arn:aws:iam::123456789012:role/MyRole"

3. Amazon Redshift (standard authentication)

[connections.source.redshift]
username = "myuser"
password = "mypassword"
database = "mydatabase"
host = "my-cluster.abcdef123456.us-west-2.redshift.amazonaws.com"
port = 5439
auth_method = "standard"
# Optional fields for UNLOAD strategy
# unload_s3_bucket = "my-migrations-bucket"
# unload_iam_role_arn = "arn:aws:iam::123456789012:role/MyRole"

4. Teradata (base connection)

Use teradatasql when the Worker environment provides it; otherwise set odbc_driver to the exact Teradata ODBC driver name. For extraction.strategy: "write_nos", add write_nos_location_scheme, write_nos_location_host, write_nos_location_container, and one credential mode under the same [connections.source.teradata] table—see the Teradata tab. For "tpt", install Teradata Tools and Utilities so tbuild is on PATH, or set TPT_TBUILD_EXECUTABLE; optional tpt_delimiter / tpt_max_sessions.

[connections.source.teradata]
host = "your-teradata-host.example.com"
port = 1025
database = "tpcds"
username = "your_username"
password = "your_password"
# odbc_driver = "Teradata Database ODBC Driver 17.20"  # when not using teradatasql
# dbc_name = "TDPID_ALIAS"
# authentication = "LDAP"

Platform-specific Details¶

Additional setup that applies only to certain sources. For connection and extraction options shared across platforms, see Source-platform specifics.

CREATE OR REPLACE STAGE <stage_name>
  URL = 's3://<your_bucket_name>/<path>/'
  STORAGE_INTEGRATION = <your_storage_integration>
  FILE_FORMAT = (TYPE = 'PARQUET');

CREATE OR REPLACE STAGE <stage_name>
  URL = 's3://<your_bucket_name>/<path>/'
  CREDENTIALS = (AWS_KEY_ID = '<your_aws_key_id>' AWS_SECRET_KEY = '<your_aws_secret_key>')
  FILE_FORMAT = (TYPE = 'PARQUET');

LIST @<stage_name>;

PUT file:///<local_test_file_path> @<stage_name>;
LIST @<stage_name>;

DESCRIBE STAGE <stage_name>;