Develop with a local IDE¶

You can run Spark workloads interactively from Jupyter Notebooks, VS Code, IntelliJ, or any Python/Java/Scala interface without needing to manage a Spark cluster. The workloads run on the Snowflake infrastructure.

To connect, install the snowpark-connect Python package, which is required for all languages (Python, Java, and Scala). For Java and Scala projects, also add the snowpark-connect-java-client Maven dependency. For establishing a connection, use a TOML connection file. This approach handles server lifecycle, authentication, and session management automatically.

You can run your workload interactively (for example, from a notebook, an IDE, or the PySpark shell), or you can run a packaged Java or Scala application JAR directly with the snowpark-connect-execute-jar command-line tool. For more information, see Running Java or Scala workloads.

Prerequisites¶

• You have a Snowflake account with access to Snowpark Connect for Spark.
• Python 3.10 or later (earlier than 3.13) is installed. Confirm your version by running python3 --version.
• Ensure that your Java and Python installations use the same CPU architecture. For example, if Python is arm64, install an arm64 build of Java (not x86_64).

Connection configuration¶

Snowpark Connect for Spark connects to Snowflake using a TOML connection file. You can create this file manually or by using Snowflake CLI.

If you have Snowflake CLI installed, you can use it to define a connection. Otherwise, you can manually write connection parameters in a config.toml file.

Add a connection by using Snowflake CLI¶

You can use Snowflake CLI to add connection properties that Snowpark Connect for Spark uses to connect to Snowflake. Your changes are saved to a config.toml file.

1. Run the following command to add a connection:

   Copy code

   snow connection add
   

2. Follow the prompts to define a connection.

   Specify spark-connect as the connection name.

   This command adds a connection to your config.toml file:

   Copy code

   [connections.spark-connect]
   host = "example.snowflakecomputing.com"
   port = 443
   account = "example"
   user = "test_example"
   password = "password"
   protocol = "https"
   warehouse = "example_wh"
   database = "example_db"
   schema = "public"
   

3. Confirm the connection works:

   Copy code

   snow connection list
   snow connection test --connection spark-connect
   

Add a connection manually¶

You can write or update a connections.toml file so that your code can connect to Snowpark Connect for Spark on Snowflake.

1. Ensure that the file permissions allow only the owner to read and write:

   Copy code

   chmod 0600 ~/.snowflake/connections.toml
   

2. Edit the file to contain a [spark-connect] connection with your specifics:

   Copy code

   [spark-connect]
   host="my_snowflake_account.snowflakecomputing.com"
   account="my_snowflake_account"
   user="my_user"
   password="&&&&&&&&"
   warehouse="my_wh"
   database="my_db"
   schema="public"
   

Install Snowpark Connect for Spark¶

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade --force-reinstall 'snowpark-connect[jdk]'
pip install pyspark==3.5.6

Start a session and run code¶

Once you have Snowpark Connect for Spark installed and an authenticated connection in place, start a session and run Spark code.

from snowflake import snowpark_connect
spark = snowpark_connect.init_spark_session()

from pyspark.sql import Row

df = spark.createDataFrame([
    Row(id=1, name="Alice", age=25),
    Row(id=2, name="Bob", age=30),
    Row(id=3, name="Charlie", age=35),
])

df.show()
df.filter(df.age > 28).show()
print(df.count())

import com.snowflake.snowpark_connect.client.SnowparkConnectSession;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;

public class SnowparkConnectExample {
    public static void main(String[] args) {
        SparkSession spark = SnowparkConnectSession.builder()
            .pythonVenv("/path/to/scos-venv")
            .appName("My App")
            .getOrCreate();

        Dataset<Row> df = spark.sql("SELECT 1 AS id, 'Alice' AS name, 25 AS age "
            + "UNION ALL SELECT 2, 'Bob', 30 "
            + "UNION ALL SELECT 3, 'Charlie', 35");

        df.show();
        df.filter("age > 28").show();
        System.out.println(df.count());

        spark.stop();
    }
}

mvn compile exec:java -Dexec.mainClass="SnowparkConnectExample"

SparkSession spark = SnowparkConnectSession.builder()
    .appName("My App")
    .config("snowpark.connect.scala.version", "2.13")
    .getOrCreate();

import com.snowflake.snowpark_connect.client.SnowparkConnectSession

object SnowparkConnectExample {
  def main(args: Array[String]): Unit = {
    val spark = SnowparkConnectSession.builder()
      .pythonVenv("/path/to/scos-venv")
      .appName("My App")
      .getOrCreate()

    try {
      import spark.implicits._

      val df = spark.createDataFrame(Seq(
        (1, "Alice", 25),
        (2, "Bob", 30),
        (3, "Charlie", 35)
      )).toDF("id", "name", "age")

      df.show()
      df.filter($"age" > 28).show()
      println(df.count())

    } finally {
      spark.stop()
    }
  }
}

sbt "runMain SnowparkConnectExample"

val spark = SnowparkConnectSession.builder()
  .appName("My App")
  .config("snowpark.connect.scala.version", "2.13")
  .getOrCreate()

Running Java or Scala workloads¶

For Java and Scala workloads, Snowpark Connect for Spark provides two approaches. Choose the one that matches how you build and run your application.

snowpark-connect-java-client library: Use this approach for interactive development, when you build and run your application from source in an IDE, Jupyter notebook, or REPL. You add the client as a Maven or sbt dependency so that it becomes part of your build. The library starts and manages a local Snowpark Connect for Spark server for you and ties the server’s lifecycle to your application’s JVM process, so your session connects to Snowflake without additional setup while you develop. For the API reference and examples, see Snowpark Connect for Spark Java/Scala client reference.

snowpark-connect-execute-jar CLI: Use this approach to run an application that’s already packaged as a JAR, without modifying or rebuilding it. The command-line tool runs your existing JAR against Snowflake, managing the local server and the connection for you and shutting everything down when the application exits. Because it requires no code changes, it’s a good fit for running built artifacts, scheduled batch jobs, or applications that you can’t or don’t want to recompile. For prerequisites, command options, and examples, see Snowpark Connect for Spark execute-jar CLI reference.

Common installation issues¶

Use the following checks to resolve common Snowpark Connect for Spark installation issues.

• Ensure that Java and Python are based on the same architecture.

• Use the most recent Snowpark Connect for Spark package, as described in Install Snowpark Connect for Spark.

• Confirm that the python command with PySpark code is working correctly for local execution without Snowflake connectivity.

  For example, execute a command such as the following:

  Copy code

  python your_pyspark_file.py