Develop with a local IDE¶

You can run Spark workloads interactively from Jupyter Notebooks, VS Code, IntelliJ, or any Python/Java/Scala interface without needing to manage a Spark cluster. The workloads run on the Snowflake infrastructure.

To connect, install the snowpark-connect Python package, which is required for all languages (Python, Java, and Scala). For Java and Scala projects, also add the snowpark-connect-java-client Maven dependency. For establishing a connection, use a TOML connection file. This approach handles server lifecycle, authentication, and session management automatically.

You can run your workload interactively (for example, from a notebook, an IDE, or the PySpark shell), or you can run a packaged Java or Scala application JAR directly with the snowpark-connect-execute-jar command-line tool. For more information, see Running Java or Scala workloads.

Prerequisites¶

• You have a Snowflake account with access to Snowpark Connect for Spark.
• Python 3.10 or later (earlier than 3.13) is installed. Confirm your version by running python3 --version.
• Ensure that your Java and Python installations use the same CPU architecture. For example, if Python is arm64, install an arm64 build of Java (not x86_64).

Connection configuration¶

Snowpark Connect for Spark connects to Snowflake using a TOML connection file. You can create this file manually or by using Snowflake CLI.

If you have Snowflake CLI installed, you can use it to define a connection. Otherwise, you can manually write connection parameters in a config.toml file.

Add a connection by using Snowflake CLI¶

You can use Snowflake CLI to add connection properties that Snowpark Connect for Spark uses to connect to Snowflake. Your changes are saved to a config.toml file.

1. Run the following command to add a connection:

   Copy code

   snow connection add
   

2. Follow the prompts to define a connection.

   Specify spark-connect as the connection name.

   This command adds a connection to your config.toml file:

   Copy code

   [connections.spark-connect]
   host = "example.snowflakecomputing.com"
   port = 443
   account = "example"
   user = "test_example"
   password = "password"
   protocol = "https"
   warehouse = "example_wh"
   database = "example_db"
   schema = "public"
   

3. Confirm the connection works:

   Copy code

   snow connection list
   snow connection test --connection spark-connect
   

Add a connection manually¶

You can write or update a connections.toml file so that your code can connect to Snowpark Connect for Spark on Snowflake.

1. Ensure that the file permissions allow only the owner to read and write:

   Copy code

   chmod 0600 ~/.snowflake/connections.toml
   

2. Edit the file to contain a [spark-connect] connection with your specifics:

   Copy code

   [spark-connect]
   host="my_snowflake_account.snowflakecomputing.com"
   account="my_snowflake_account"
   user="my_user"
   password="&&&&&&&&"
   warehouse="my_wh"
   database="my_db"
   schema="public"
   

Install Snowpark Connect for Spark¶

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade --force-reinstall 'snowpark-connect[jdk]'
pip install pyspark==3.5.6

Start a session and run code¶

Once you have Snowpark Connect for Spark installed and an authenticated connection in place, start a session and run Spark code.

from snowflake import snowpark_connect
spark = snowpark_connect.init_spark_session()

from pyspark.sql import Row

df = spark.createDataFrame([
    Row(id=1, name="Alice", age=25),
    Row(id=2, name="Bob", age=30),
    Row(id=3, name="Charlie", age=35),
])

df.show()
df.filter(df.age > 28).show()
print(df.count())

import com.snowflake.snowpark_connect.client.SnowparkConnectSession;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;

public class SnowparkConnectExample {
    public static void main(String[] args) {
        SparkSession spark = SnowparkConnectSession.builder()
            .pythonVenv("/path/to/scos-venv")
            .appName("My App")
            .getOrCreate();

        Dataset<Row> df = spark.sql("SELECT 1 AS id, 'Alice' AS name, 25 AS age "
            + "UNION ALL SELECT 2, 'Bob', 30 "
            + "UNION ALL SELECT 3, 'Charlie', 35");

        df.show();
        df.filter("age > 28").show();
        System.out.println(df.count());

        spark.stop();
    }
}

mvn compile exec:java -Dexec.mainClass="SnowparkConnectExample"

SparkSession spark = SnowparkConnectSession.builder()
    .appName("My App")
    .config("snowpark.connect.scala.version", "2.13")
    .getOrCreate();

import com.snowflake.snowpark_connect.client.SnowparkConnectSession

object SnowparkConnectExample {
  def main(args: Array[String]): Unit = {
    val spark = SnowparkConnectSession.builder()
      .pythonVenv("/path/to/scos-venv")
      .appName("My App")
      .getOrCreate()

    try {
      import spark.implicits._

      val df = spark.createDataFrame(Seq(
        (1, "Alice", 25),
        (2, "Bob", 30),
        (3, "Charlie", 35)
      )).toDF("id", "name", "age")

      df.show()
      df.filter($"age" > 28).show()
      println(df.count())

    } finally {
      spark.stop()
    }
  }
}

sbt "runMain SnowparkConnectExample"

val spark = SnowparkConnectSession.builder()
  .appName("My App")
  .config("snowpark.connect.scala.version", "2.13")
  .getOrCreate()

Running Java or Scala workloads¶

For Java and Scala workloads, Snowpark Connect for Spark provides two approaches. Choose the one that matches how you build and run your application.

snowpark-connect-java-client library: Use this approach for interactive development, when you build and run your application from source in an IDE, Jupyter notebook, or REPL. You add the client as a Maven or sbt dependency so that it becomes part of your build. The library starts and manages a local Snowpark Connect for Spark server for you and ties the server’s lifecycle to your application’s JVM process, so your session connects to Snowflake without additional setup while you develop. For the API reference and examples, see Snowpark Connect for Spark Java/Scala client reference.

snowpark-connect-execute-jar CLI: Use this approach to run an application that’s already packaged as a JAR, without modifying or rebuilding it. The command-line tool runs your existing JAR against Snowflake, managing the local server and the connection for you and shutting everything down when the application exits. Because it requires no code changes, it’s a good fit for running built artifacts, scheduled batch jobs, or applications that you can’t or don’t want to recompile. For prerequisites, command options, and examples, see Snowpark Connect for Spark execute-jar CLI reference.

Common installation issues¶

Use the following checks to resolve common Snowpark Connect for Spark installation issues.

• Ensure that Java and Python are based on the same architecture.

• Use the most recent Snowpark Connect for Spark package, as described in Install Snowpark Connect for Spark.

• Confirm that the python command with PySpark code is working correctly for local execution without Snowflake connectivity.

  For example, execute a command such as the following:

  Copy code

  python your_pyspark_file.py
  

Connect directly to Snowflake’s Spark Connect endpoint¶

You can connect to Snowflake’s hosted Spark Connect endpoint using standard, off-the-shelf Spark client packages such as PySpark or Spark clients for Java and Scala. You don’t need to install any Snowflake-specific packages.

With this approach, all Spark processing runs on Snowflake’s infrastructure. Your client sends Spark Connect protocol messages directly to Snowflake, which executes the workload and returns results. Authentication uses programmatic access tokens (PATs).

This option is useful when you want to:

• Avoid installing Snowflake-specific packages in your environment.
• Use your existing Spark tooling (Jupyter, VS Code, terminals) with Snowflake compute and governance.
• Simplify dependency management by relying only on the standard PySpark package.

Step 1: Install required packages¶

Install the Spark Connect client for your language. You don’t need to install any Snowflake packages.

pip install "pyspark[connect]>=3.5.0,<4"

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-connect-client-jvm_2.12</artifactId>
    <version>3.5.6</version>
</dependency>

libraryDependencies += "org.apache.spark" %% "spark-connect-client-jvm" % "3.5.6"

javaOptions ++= Seq(
  "--add-opens=java.base/java.nio=ALL-UNNAMED"
)

Step 2: Set up authentication¶

1. Generate a programmatic access token (PAT).

   For more information, see the following topics:

   • Using programmatic access tokens for authentication
   • ALTER USER … ADD PROGRAMMATIC ACCESS TOKEN (PAT)

   The following example adds a PAT named TEST_PAT for the user sysadmin and sets the expiration to 30 days.

   Copy code

   ALTER USER ADD PAT TEST_PAT ROLE_RESTRICTION = sysadmin DAYS_TO_EXPIRY = 30;
   

2. Find your Snowflake Spark Connect host URL.

   Run the following SQL in Snowflake to find the hostname for your account:

   Copy code

   SELECT t.VALUE:type::VARCHAR as type,
       t.VALUE:host::VARCHAR as host,
       t.VALUE:port as port
     FROM TABLE(FLATTEN(input => PARSE_JSON(SYSTEM$ALLOWLIST()))) AS t WHERE type = 'SNOWPARK_CONNECT';
   

Step 3: Connect and run Spark code¶

Connect to the Snowflake Spark Connect endpoint using the host URL and PAT from the previous steps.

from pyspark.sql import SparkSession
import urllib.parse

# Replace with your actual PAT.
pat = urllib.parse.quote("<pat>", safe="")

# Replace with your Snowpark Connect host from the above SQL query.
snowpark_connect_host = ""

# Define database/schema/warehouse for your Spark session (recommended).
# Otherwise, values are resolved from your default_namespace and default_warehouse.
db_name = urllib.parse.quote("TESTDB", safe="")
schema_name = urllib.parse.quote("TESTSCHEMA", safe="")
warehouse_name = urllib.parse.quote("TESTWH", safe="")

spark = SparkSession.builder.remote(
    f"sc://{snowpark_connect_host}/;token={pat};token_type=PAT"
    f";database={db_name};schema={schema_name};warehouse={warehouse_name}"
).getOrCreate()

from pyspark.sql import Row

df = spark.createDataFrame([
    Row(id=1, name="Alice", age=25),
    Row(id=2, name="Bob", age=30),
    Row(id=3, name="Charlie", age=35),
])

df.show()
df.filter(df.age > 28).show()
print(df.count())

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import java.net.URLEncoder;

public class ServerSideConnectExample {
    public static void main(String[] args) throws Exception {
        String pat = URLEncoder.encode("<pat>", "UTF-8");
        String snowparkConnectHost = "";

        String dbName = URLEncoder.encode("TESTDB", "UTF-8");
        String schemaName = URLEncoder.encode("TESTSCHEMA", "UTF-8");
        String warehouseName = URLEncoder.encode("TESTWH", "UTF-8");

        SparkSession spark = SparkSession.builder()
            .remote("sc://" + snowparkConnectHost + "/;token=" + pat + ";token_type=PAT"
                + ";database=" + dbName + ";schema=" + schemaName
                + ";warehouse=" + warehouseName)
            .getOrCreate();

        Dataset<Row> df = spark.sql("SELECT 1 AS id, 'Alice' AS name, 25 AS age "
            + "UNION ALL SELECT 2, 'Bob', 30 "
            + "UNION ALL SELECT 3, 'Charlie', 35");

        df.show();
        df.filter("age > 28").show();
        System.out.println(df.count());

        spark.stop();
    }
}

mvn compile exec:java -Dexec.mainClass="ServerSideConnectExample"

import org.apache.spark.sql.SparkSession
import java.net.URLEncoder

object ServerSideConnectExample {
  def main(args: Array[String]): Unit = {
    val pat = URLEncoder.encode("<pat>", "UTF-8")
    val snowparkConnectHost = ""

    val dbName = URLEncoder.encode("TESTDB", "UTF-8")
    val schemaName = URLEncoder.encode("TESTSCHEMA", "UTF-8")
    val warehouseName = URLEncoder.encode("TESTWH", "UTF-8")

    val spark = SparkSession.builder()
      .remote(
        s"sc://$snowparkConnectHost/;token=$pat;token_type=PAT" +
        s";database=$dbName;schema=$schemaName;warehouse=$warehouseName"
      )
      .getOrCreate()

    try {
      import spark.implicits._

      val df = spark.createDataFrame(Seq(
        (1, "Alice", 25),
        (2, "Bob", 30),
        (3, "Charlie", 35)
      )).toDF("id", "name", "age")

      df.show()
      df.filter($"age" > 28).show()
      println(df.count())

    } finally {
      spark.stop()
    }
  }
}

sbt "runMain ServerSideConnectExample"