description:

Using the spark-migration Snowflake CoCo skill to migrate PySpark code to the Snowpark Python API with the SMA CLI

Spark to Snowpark API with the Snowflake CoCo migration skill¶

You can migrate PySpark code to the native Snowpark Python API (snowflake.snowpark, snake_case) by using the spark-migration skill in Snowflake CoCo. This path runs the Snowpark Migration Accelerator (SMA) CLI for conversion, then continues through a post-conversion pipeline (EWI dashboard, optional fixes, stage conversion, and validation).

This page covers the Snowpark API path (the bundled snowpark-api sub-skill). The default and recommended path is Snowpark Connect, which preserves the PySpark API surface. For a phase-by-phase Snowpark Connect walkthrough, see Migrating to Snowpark Connect walkthrough.

When this path runs¶

The Snowpark API path is an explicit opt-in. The skill routes here only when you clearly ask for it, for example:

• Phrases such as “SMA”, “SMA CLI”, “run sma”, “Snowpark API”, or a snowflake.snowpark rewrite
• Work on already-migrated SMA output (“open sma dashboard”, “fix ewis”, “run stage conversion”, “resume dvp”)
• An existing project config with conversion_type set to snowpark-api

If the intent is ambiguous, the skill asks which path you want. It never silently defaults to the Snowpark API path.

Scala isn’t supported on this path. If Scala is detected, you’re prompted to switch to Snowpark Connect, which supports Scala.

How the path is organized¶

Inside spark-migration, the Snowpark API work is handled by bundled sub-skills (loaded from the skill directory, not as separately registered skills). At a high level:

spark-migration/
├── SKILL.md                              Parent router
├── configurations/{project}.json         Per-project state
├── config.json                           Global state (including sma_cli_path)
├── snowflake-notebook-migration/         Shared notebook converter
└── snowpark-api/
    ├── migrate-pyspark-to-snowpark-api/  Fresh SMA run (M1–M6)
    ├── validate-pyspark-to-snowpark-api/ Post-conversion pipeline (V0–V8)
    ├── sma-dashboard-generator/
    ├── stage-conversion/
    └── dvp/                              Validation orchestrator and helpers

Intent routing inside the path¶

After the parent skill delegates to the Snowpark API path:

1. Language is detected (imports, extensions, build files). You’re asked if it’s ambiguous.
2. Python with a migrate intent runs the SMA conversion steps (M1–M6), then hands off to the post-conversion pipeline.
3. Python with an already-migrated intent skips the SMA run and enters the post-conversion pipeline (V0–V8).
4. Scala stops and offers a switch to Snowpark Connect.

Control doesn’t return to the parent until the pipeline completes or you abort.

Configuration¶

Every project has a JSON config at {skill_directory}/configurations/{project_name}.json, managed by the skill’s config helper (don’t edit these files by hand). The Snowpark API path reads a namespaced slice of that config.

Shared keys (owned by the parent skill): project_name (required; also passed as -p to the SMA CLI), input_folder (-i), output_folder (-o), email (-e), company (-c), conversion_type, migration_status (migrate or already_migrated), and run_notebook_migration.

Snowpark API keys (persisted and used as gates for later steps):

The Snowpark API path uses these keys:

Global key: sma_cli_path in config.json: absolute path to the sma binary. It’s auto-detected or you supply it; the skill validates that the path is executable.

Gated steps use the same rules: no skips the step; yes uses saved values without re-asking; unset asks once, then persists. Answers aren’t re-asked on later runs.

What to expect, phase by phase¶

Progress lines are timestamped. Interactive questions appear only for unset config keys.

Phase 0: Prerequisites (parent)¶

• Git: installed if missing when possible; hard stop if unavailable.
• SMA CLI (this path only): the skill looks for sma on the PATH and in common install locations. If it isn’t found, you’re asked for the absolute path (typically under the SMA install’s orchestrator/sma). The pipeline doesn’t continue without a verified executable.

Phase 1: Config load and dispatch (parent)¶

A start time is recorded for the final summary. You’re offered existing project configs or create a new one (project name is required up front). You can review and accept or edit settings, then routing follows migration_status and conversion_type.

Phase 2: SMA conversion (M1–M6)¶

The migrate sub-skill runs these steps:

Flag derivation: enable_jupyter_conversion=no adds -x; a non-default sql_flavor adds -f {flavor}; generate_checkpoints=no adds -d. License acceptance (-y) is always passed.

Failure handling: Error lines or a non-zero exit stop the run and surface the error. If SMA produces no new output for more than about 10 minutes, you’re asked whether to wait or abort. Common fixes: make the binary executable, check for another sma process, and confirm the input path is a directory rather than a single notebook.

Phase 3: Post-conversion pipeline (V0–V8)¶

The validator runs these steps after conversion (or when you start from already-migrated SMA output):

SMA output layouts¶

The SMA CLI version determines the on-disk layout. The workload root is the directory that contains Output/ and Reports/.

Layouts resolve as follows:

Detection order: .snowma → v1; sma-output/ → v2; Conversion_Snowpark* with sma-code-process-* children → v3; legacy Conversion-* → v1; otherwise the skill asks you.

Two invariants:

• Work stays inside the resolved folder. The skill doesn’t copy Output/, Reports/, or Logs/ upward.
• A missing Reports/Issues.csv after resolution means conversion failed silently. The skill stops, points you at {output}/Logs/, and asks about re-running.

Results and artifacts¶

After a successful run, look for these artifacts:

Final summary¶

The run ends with a fixed-format summary table. Top-level rows cover the SMA CLI conversion, dashboard generator, notebook migration, EWI fixer, stage conversion, DVP orchestrator (with sub-steps through test run and test fixer), and reopening the dashboard. Each row is marked Done, Skipped, or Failed with counts where applicable.

The summary also includes the output location, the two git branches, wall-clock duration, the dashboard reopen command, and next steps such as re-running tests with pytest {output}/dvp/03-tests/source -v, remaining EWI fixes, and reviewing code in Output/.

The already-migrated variant changes only the first row (no fresh SMA CLI run).

Entry points¶

These prompts route into the Snowpark API path:

Invoking the skill¶

/spark-migration migrate [path] to snowpark-api using the SMA CLI

/spark-migration I already ran SMA on [path]; continue with the Snowpark API post-conversion pipeline

/spark-migration open the SMA dashboard for [path]

Plain-language prompts that mention SMA or Snowpark API also route here when intent is clear.

Known stopping points¶

• SMA CLI can’t be validated → detection runs again; the pipeline doesn’t proceed unverified
• SMA non-zero exit or Error: lines → error is shown as-is; retry only if you choose
• SMA silent for more than about 10 minutes → you’re asked to wait or abort (no automatic retry)
• Output layout can’t be resolved, or Issues.csv is missing → check Logs/; treat as a failed conversion
• Dirty git repo and you choose abort → stop for manual resolution
• Dashboard generator failure → later steps wait until that’s fixed
• Required bundled skill files missing → stop and reinstall the spark-migration skill

Related documentation¶

• Migrating with Snowflake CoCo: spark-migration overview
• Spark to Snowpark Connect: default conversion path
• Migrating to Snowpark Connect walkthrough: phase walkthrough for Snowpark Connect
• Snowpark Migration Accelerator: SMA install and getting started
• Snowpark Developer Guide: Snowpark Python API