Data migration configuration reference¶

This page documents the workflow YAML and Worker TOML settings for Data Migration. For platform-specific prerequisites, auth methods, extraction strategies, and data type mappings, see the per-platform pages linked from Data migration.

For SnowConvert AI CLI commands that generate and submit these configs, see Manual Migration with SnowConvert AI CLI: Data migration.

Workflow configuration reference¶

The Data Migration Workflow configuration file is a YAML file. The CLI and Snowflake AIM Agent for Data Warehouses accept .yaml or .yml files.

Top-level object¶

When defaultTableConfiguration is present, each object in tables is merged with those defaults: shared fields apply to every table unless the same field is set again on a specific table entry.

TableConfiguration model¶

Defines the settings for migrating a single table.

QueryModifiers model¶

SourceTargetIdentifier model¶

Additional target properties¶

The following optional fields apply only to the target object.

Iceberg configuration (target.icebergConfig)¶

Used when target.tableType is "iceberg". Account setup follows Snowflake’s Apache Iceberg™ tables documentation.

Partition size¶

Partition size is controlled at the TableConfiguration level with two flat, mutually exclusive fields: targetPartitionSizeMb and targetPartitionSizeRows. When both are omitted, the orchestrator uses auto sizing.

Specify at most one of the two fields; setting both on the same table is a configuration error.

Load segmentation¶

When a large number of files are staged for a single partition (common with Redshift UNLOAD), the orchestrator can split the COPY INTO into multiple parallel statements, each targeting a subset of files. All resulting statements fan in to the same successor task.

When loadSegmentation is omitted, a single COPY INTO loads all files in the partition (the default). Files larger than the target size get their own segment, and Snowflake’s 1,000-file FILES limit is enforced per statement.

    loadSegmentation:
      targetSegmentSizeMb: 5000

ExtractionStrategy model¶

See Extraction strategies for YAML examples and platform availability.

SynchronizationStrategy model¶

Affinity¶

By specifying an affinity for a workflow, you indicate that specific Workers should execute that workflow. Matching rules:

• A task without affinity is picked up by any Worker.
• A Worker without affinity picks up any task.
• A task with a given affinity is not picked up by a Worker with a different affinity.

Extraction strategies¶

Each table can set extraction.strategy (or inherit from defaultTableConfiguration.extraction). Worker TOML holds connection and platform-specific options (UNLOAD credentials, WRITE_NOS location, TPT tuning).

Regular (default):

tables:
  - source:
      databaseName: MY_DB
      schemaName: sales
      tableName: orders
    target:
      databaseName: TARGET_DB
      schemaName: sales
      tableName: orders
    columnNamesToPartitionBy:
      - order_id

Redshift unload:

    extraction:
      strategy: unload
      externalStage: TARGET_DB.PUBLIC.S3_EXTERNAL_STAGE

Teradata write_nos:

    extraction:
      strategy: write_nos
      externalStage: TARGET_DB.SALES.MY_CLOUD_STAGE

Oracle dbms_cloud:

    extraction:
      strategy: dbms_cloud
      externalStage: TARGET_DB.DATA_MIGRATION.ORA_EXPORT_STAGE

Teradata tpt:

    extraction:
      strategy: tpt

Worker configuration¶

The Worker configuration file uses TOML format. For cloud data migration, set selected_task_source to "snowflake_stored_procedure" and provide a matching [task_source.snowflake_stored_procedure] section.

Example minimal Worker config:

selected_task_source = "snowflake_stored_procedure"

[task_source.snowflake_stored_procedure]
connection_name = "my-snowflake"

[application]
max_parallel_tasks = 4
task_fetch_interval = 30

[connections.source.sqlserver]
username = "username"
password = "password"
database = "database_name"
host = "127.0.0.1"
port = 1433

[connections.target.snowflake_connection_name]
connection_name = "my-snowflake"

Platform-specific [connections.source.*] examples live on each Migrating Data from … page.

Anti-locking and query modifiers¶

User-configured anti-locking hints are off by default. Unless you set queryModifiers, AIM DMV runs plain source queries. Modifiers apply to source queries only, never the Snowflake target.

Per-platform behavior¶

Pros and cons¶

Workflow example¶

Set a default on defaultTableConfiguration and override per table when needed:

defaultTableConfiguration:
  queryModifiers:
    objectModifier: " WITH (NOLOCK)"
tables:
  - source:
      databaseName: MY_DB
      schemaName: sales
      tableName: orders
    target:
      databaseName: TARGET_DB
      schemaName: sales
      tableName: orders
    queryModifiers:
      objectModifier: " WITH (NOLOCK)"

Worker TOML example¶

You can also set query modifiers in Worker TOML under the source connection:

[connections.source.sqlserver.query_modifiers]
object_modifier = " WITH (NOLOCK)"
select_modifier = ""

Related content¶

• Data Migration & Validation overview
• Data migration
• The SNOWCONVERT_AI database
• Data validation configuration reference
• Manual Migration with SnowConvert AI CLI: Data migration