Run Spark workloads from Snowflake Notebooks

You can run Spark workloads interactively from Snowflake Notebooks without needing to manage a Spark cluster. The workloads run on the Snowflake infrastructure.

To use Snowflake Notebooks as a client for developing Spark workloads to run on Snowflake:

  1. Launch Snowflake Notebooks.

  2. Within the notebook, start a Spark session.

  3. Write PySpark code to load, transform, and analyze data—such as to filter high-value customer orders or aggregate revenue.

Use a Snowflake Notebook that runs on a warehouse

For more information about Snowflake Notebooks, see Create a notebook.

  1. Create a Snowflake Notebook by completing the following steps:

    1. Sign in to Snowsight.

    2. From the navigation menu, select + Create » Notebook » New Notebook.

    3. In the Create notebook dialog, enter a name, database, and schema for the new notebook.

      For more information, see Create a notebook.

    4. For Runtime, select Run on warehouse.

    5. For Runtime version, select Snowflake Warehouse Runtime 2.0.

      When you select version 2.0, you ensure that you have the dependency support you need, including Python 3.10. For more information, see Notebook runtimes.

    6. For Query warehouse and Notebook warehouse, select warehouses for running query code and kernel and Python code, as described in Create a notebook.

    7. Select Create.

    8. In the notebook you created, under Packages, ensure that you have the following packages listed to support code in your notebook:

      • Python, version 3.10 or later

      • snowflake-dataframe-processor, latest version

        If you need to add these packages, use the following steps:

        1. Under Anaconda Packages, type the packages name in the search box.

        2. Select the package name.

        3. Select Save.

  2. To connect to the Snowpark Connect for Spark server and test the connection, copy the following code and paste it in the Python cell of the notebook you created:

    from snowflake import snowpark_connect

spark = snowpark_connect.server.init_spark_session()
df = spark.sql("show schemas").limit(10)
df.show()
    Copy
Language: English