Snowpark Migration Accelerator: Walkthrough Setup¶

SMA Application¶

The Snowpark Migration Accelerator (SMA) helps developers convert their PySpark and Spark Scala applications to run on Snowflake. It automatically detects Spark API calls in your Python or Scala code and transforms them into equivalent Snowpark API calls. This guide will demonstrate basic SMA functionality by analyzing sample Spark code and showing how it assists with migration projects.

During the initial assessment phase, Snowpark Migration Accelerator (SMA) examines your source code and builds a detailed model that captures all the functionality in your code. Based on this analysis, SMA creates several reports, including a detailed assessment report that we’ll review in this walkthrough. These reports help you understand how ready your code is for migration to Snowpark and estimate the effort needed for the transition. We’ll look at these findings in more detail as we continue through this lab.

Sample Codebase¶

This guide uses Python code examples to demonstrate the migration process. We have selected two publicly available sample codebases from third-party Git repositories as unbiased, real-world examples. You can access these codebases at:

• PySpark Data Engineering Examples: https://github.com/spark-examples/pyspark-examples

• Apache Spark Machine Learning Examples: https://github.com/apache/spark/tree/master/examples/src/main/python

To analyze codebases using the Snowpark Migration Accelerator (SMA), follow these steps:

1. Download the codebases as zip files from GitHub. You can find instructions on how to do this in the GitHub documentation.

2. Create separate folders on your computer for each codebase.

3. Extract each zip file into its designated folder, as shown in the image below:

These sample codebases demonstrate how SMA evaluates Spark API references to calculate the Spark API Readiness Score. Let’s look at two scenarios:

1. A codebase that received a high score, indicating it is highly compatible with Snowpark and ready for migration

2. A codebase that received a low score, indicating it requires additional review and potential modifications before migration

While the readiness score provides valuable insight, it should not be the only factor considered when planning a migration. A comprehensive evaluation of all aspects is necessary for both high and low scoring assessments to ensure a successful migration.

After unzipping the directories, SMA will analyze only files that use supported code formats and notebook formats. These files are checked for references to Spark API and other Third Party APIs. To see which file types are supported, please check the list here.

Throughout the rest of this walkthrough, we will analyze how these two codebases execute.