Submit Spark jobs on Snowflake¶

Run your Spark applications as batch jobs on Snowflake. You submit a job, Snowflake runs it on Snowflake compute, and you get back an ID that you use to track, monitor, and cancel the run.

Overview¶

A Spark job runs your packaged Spark application (Scala or Python) as a single batch run. When you submit a job, you provide:

• The application: a .jar (Scala/Java) or .py (Python) file on a Snowflake stage.
• The entrypoint: the main class or file to run.
• Arguments: optional command-line arguments passed to your application.
• A specification: the runtime version, language, dependencies, and other settings for the run.

Snowflake runs the job on warehouse compute and returns a job ID. You use that ID to monitor and manage the run.

You can submit a Spark job two ways:

• Submit a job with the REST API: set asyncExec=true to submit asynchronously and poll for status, or omit it to call the API synchronously and wait for the run to finish. This is a recommended approach for orchestrating from data pipelines and CI/CD systems such as Airflow, GitHub Actions, GitLab CI, and Jenkins, or from external control planes.
• Submit a job with SQL: a synchronous SQL statement that submits the job and waits for it to finish. Because it’s SQL, it integrates natively with Snowflake tasks, so you can build scheduled Spark pipelines on Snowflake.

You provide the job definition inline with each submission. To reuse a stored job definition instead, you can persist it.

Supported languages and runtimes¶

The runtime is selected with compute_options.runtime_version. The current runtime version is "1.29", which provides Python 3.11 for Python jobs and Scala 2.12 or 2.13 for Scala jobs.

Prerequisites¶

• A Snowflake account enrolled in the private preview, and a warehouse to run your jobs on.
• A Snowflake stage where you can upload your application files and dependencies.
• A packaged Spark application. See Develop your Spark application.

Develop your Spark application¶

Develop your application with the standard Spark APIs (SparkSession and DataFrames). For how to develop, package, and configure your application, see the Snowflake documentation for running Spark on Snowflake:

• Run Spark workloads on Snowflake
• Spark application examples

Once your application is packaged, upload it to a Snowflake stage so you can submit it. For a Scala or Java application, package it as a fat JAR and note the fully-qualified main class:

PUT file://target/scala-2.12/my-app_2.12-1.0.0.jar @my_db.my_schema.my_stage/spark_jobs/ AUTO_COMPRESS=FALSE OVERWRITE=TRUE;

For a Python application, upload your entrypoint file (and any supporting files):

PUT file://job.py @my_db.my_schema.my_stage/python_jobs/ AUTO_COMPRESS=FALSE OVERWRITE=TRUE;

Submit a job with the REST API¶

Submit a Spark job by posting the job definition to the code bundle executions endpoint. This is the recommended option for orchestrating Spark job pipelines from external orchestrators such as Apache Airflow and other clients such as CI/CD and custom UIs.

Endpoint¶

POST /api/v2/code-bundle-executions

By default the call is synchronous and returns when the run finishes. To submit asynchronously, set the asyncExec query parameter to true; the call returns immediately with a job ID that you poll for status.

POST /api/v2/code-bundle-executions?asyncExec=true

Request headers¶

Request headers follow the standard Snowflake REST API specification. Authenticate with a key-pair JWT (shown here) or another supported method, such as OAuth or a programmatic access token. The X-Snowflake-Warehouse header selects the warehouse that your job runs on.

curl -X POST \
  "https://<account_identifier>.snowflakecomputing.com/api/v2/code-bundle-executions?asyncExec=true&requestId=5f3f8b16-1d2c-4b9a-9c2a-7c0d2a3b4c5d" \
  -H "Authorization: Bearer ${SNOWFLAKE_TOKEN}" \
  -H "X-Snowflake-Authorization-Token-Type: KEYPAIR_JWT" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -H "User-Agent: myApplicationName/1.0" \
  -H "X-Snowflake-Role: MY_ROLE" \
  -H "X-Snowflake-Warehouse: MY_SUBMIT_WAREHOUSE" \
  -d @request-body.json

Idempotent submission¶

requestId is an optional query parameter (a client-generated UUID) that makes submission idempotent. If a submit call times out or you retry with the same requestId, Snowflake doesn’t start a duplicate run, which prevents orchestrator retries (Airflow, CI/CD) from starting the same batch twice. Idempotency is keyed only on requestId: the request body isn’t compared, so a retry that reuses a requestId with a different body is still treated as a duplicate and starts no new run.

Request body¶

The body contains the stage location of your application (from_location), the entrypoint, an optional arguments array, an optional execution_name, and the job specification: a JSON bundle object with the same structure and fields as the YAML you use in the SQL WITH SPECIFICATION clause. In the REST API, pass the specification as a JSON object, not a string. See Job specification reference for the fields.

The optional execution_name assigns a name to the run, the same as the SQL EXECUTION_NAME parameter: Snowflake stores it in CODE_BUNDLE_HISTORY and renders it into the server-side EXECUTE CODE BUNDLE statement. If you omit it, the run’s EXECUTION_NAME is empty.

Scala/Java:

{
  "from_location": "@my_db.my_schema.my_stage/spark_jobs/job.jar",
  "entrypoint": "com.example.MySparkApp",
  "arguments": ["--input", "@my_db.my_schema.my_stage/input.csv", "--partitions", "10"],
  "execution_name": "my_spark_application",
  "specification": {
    "bundle": {
      "type": "spark",
      "compute_type": "warehouse",
      "language": "scala",
      "compute_options": { "runtime_version": "1.29", "language_version": "2.12" },
      "properties": {
        "java_dependencies": {
          "jars": ["@my_db.my_schema.my_stage/source/jars/library1.jar"]
        }
      }
    }
  }
}

Python:

{
  "from_location": "@my_db.my_schema.my_stage/python_jobs/job.py",
  "entrypoint": "job.py",
  "arguments": ["--input", "@my_db.my_schema.my_stage/input.csv", "--partitions", "10"],
  "execution_name": "my_spark_application",
  "specification": {
    "bundle": {
      "type": "spark",
      "compute_type": "warehouse",
      "language": "python",
      "compute_options": { "runtime_version": "1.29" },
      "properties": {
        "python_dependencies": {
          "requirements_files": ["@my_db.my_schema.my_stage/python_jobs/requirements.txt"]
        }
      }
    }
  }
}

Response¶

An asynchronous submission (asyncExec=true) returns 202 Accepted with a job ID in the body. Use the job ID to monitor and manage the run.

HTTP/1.1 202 Accepted
Content-Type: application/json
Location: /api/v2/code-bundle-executions/01b1f2e0-0000-df4f-0000-00100006589e

{
  "code": "392604",
  "message": "Request execution in progress. Use provided Location header or result handler id to perform query monitoring and management.",
  "result_handler": "01b1f2e0-0000-df4f-0000-00100006589e",
  "job_id": "01b1f2e0-0000-df4f-0000-00100006589e"
}

Use job_id (equivalently, the ID in the Location header) as the run identifier for status and cancel calls.

Submit a job with SQL¶

Submit a Spark job with the EXECUTE CODE BUNDLE command. The command takes the entrypoint, arguments, and an optional execution name as SQL parameters; the rest of the job definition is passed inline as YAML in the WITH SPECIFICATION clause. The job inherits the role that runs the statement and runs on the warehouse set in the session. The statement is synchronous: it returns when the run finishes.

Set the session context¶

Set the warehouse, database, and schema before submitting. Your job runs on the session warehouse.

USE WAREHOUSE my_warehouse;
USE DATABASE my_db;
USE SCHEMA my_schema;

Submit a job¶

Define and submit the job in a single statement, with the application, entrypoint, and specification all passed inline.

Scala/Java:

EXECUTE CODE BUNDLE FROM '@my_db.my_schema.my_stage/spark_jobs/job.jar'
  ENTRYPOINT = 'com.example.MySparkApp'
  ARGUMENTS = ('--input', '@my_db.my_schema.my_stage/input.csv', '--partitions', '10')
  EXECUTION_NAME = 'my_spark_application'
  WITH SPECIFICATION $$
bundle:
  type: spark
  compute_type: warehouse
  language: scala
  compute_options:
    runtime_version: "1.29"
    language_version: "2.12"
  properties:
    java_dependencies:
      jars:
        - '@my_db.my_schema.my_stage/source/jars/library1.jar'
        - '@my_db.my_schema.my_stage/source/jars/library2.jar'
$$;

Python:

EXECUTE CODE BUNDLE FROM '@my_db.my_schema.my_stage/python_jobs/job.py'
  ENTRYPOINT = 'job.py'
  ARGUMENTS = ('--input', '@my_db.my_schema.my_stage/input.csv', '--partitions', '10')
  EXECUTION_NAME = 'my_spark_application'
  WITH SPECIFICATION $$
bundle:
  type: spark
  compute_type: warehouse
  language: python
  compute_options:
    runtime_version: "1.29"
  properties:
    python_dependencies:
      requirements_files:
        - '@my_db.my_schema.my_stage/python_jobs/requirements.txt'
$$;

Command parameters¶

The statement runs synchronously and returns when the run finishes. Each run is identified by its query ID. This is the same value the REST API returns as job_id and that you pass as the executionId to the status and cancel endpoints, so you can use it to monitor and manage the run.

Persist a job definition¶

If you would like to reuse a stored job definition, you can register it once and then submit it by name (for example, EXECUTE CODE BUNDLE my_spark_job ENTRYPOINT = 'com.example.MySparkApp'). The submission parameters are the same as for inline submission. For details, see Snowflake Code Bundles.

Schedule with a task¶

Because you submit the job with a SQL statement, you can wrap it in a Snowflake task to run it on a schedule. Tasks let you build and orchestrate Spark data pipelines natively in Snowflake: you can chain jobs into dependency graphs and run them on a schedule, similar to how teams orchestrate Spark workloads with Airflow, but without operating a separate scheduler.

Job specification reference¶

The job specification defines how your Spark job runs. In SQL it’s the YAML in the WITH SPECIFICATION clause; in the REST API it’s the bundle object in the request body. The following fields apply to Spark jobs. For the full set of generic configuration fields, see the code_bundle.yml reference.

Monitor and manage jobs¶

Every Spark job has one identifier, and it’s the same value everywhere you refer to the run. The query ID returned by EXECUTE CODE BUNDLE, the job_id returned by a REST submission, and the executionId you pass to the status and cancel endpoints (returned as execution_id in the status response) are all the same ID (for example, 01c51743-c819-4261-0000-5349586311aa). Use it to check status, cancel the run, and find logs. You can monitor and manage a run with SQL or the REST API, regardless of how you submitted it.

Check status¶

Check a run’s status by its job ID, using SQL or the REST API.

Using SQL:

Look up the run in the CODE_BUNDLE_HISTORY table function. Pass the job ID (returned at submission) as the QUERY_ID argument, which filters server-side:

SELECT
  CODE_BUNDLE_NAME, DATABASE_NAME, SCHEMA_NAME, ENTRYPOINT, STATUS,
  RUNTIME_STATUS_DETAILS, SQL_ERROR_CODE, ERROR_MESSAGE, BUNDLE_TYPE,
  COMPUTE_TYPE, LANGUAGE_TYPE, RUNTIME_NAME, START_TIME, END_TIME,
  QUERY_ID, EXECUTION_NAME
FROM TABLE(SNOWFLAKE.INFORMATION_SCHEMA.CODE_BUNDLE_HISTORY(QUERY_ID => '<job_id>'));

The query returns a single row for the run. For example:

{
  "CODE_BUNDLE_NAME": null,
  "DATABASE_NAME": null,
  "SCHEMA_NAME": null,
  "ENTRYPOINT": "com.example.MySparkApp",
  "STATUS": "DONE",
  "RUNTIME_STATUS_DETAILS": "",
  "SQL_ERROR_CODE": "",
  "ERROR_MESSAGE": "",
  "BUNDLE_TYPE": "SPARK",
  "COMPUTE_TYPE": "WAREHOUSE",
  "LANGUAGE_TYPE": "SCALA",
  "RUNTIME_NAME": "MY_WAREHOUSE",
  "START_TIME": "2026-07-14 10:50:00+00:00",
  "END_TIME": "2026-07-14 10:50:39+00:00",
  "QUERY_ID": "01c51743-c819-4261-0000-5349586311aa",
  "EXECUTION_NAME": "my_spark_application"
}

For a job submitted inline from a stage, CODE_BUNDLE_NAME, DATABASE_NAME, and SCHEMA_NAME are null because there’s no stored job definition. STATUS shows the run’s state, such as DONE or FAILED; when a run fails, ERROR_MESSAGE carries the failure details.

If you assigned an execution_name when submitting (with SQL or REST), you can look the run up by that name instead of the job ID: pass EXECUTION_NAME => '<execution_name>' to CODE_BUNDLE_HISTORY, or filter on the EXECUTION_NAME column.

Using the REST API:

Get the status of a run by ID. The {executionId} is the job ID returned at submission. Pass the database and schema as headers:

GET /api/v2/code-bundle-executions/{executionId}
X-Snowflake-Database: MY_DB
X-Snowflake-Schema: MY_SCHEMA

The response is an array with a single execution record. status comes from the account query history. A run is still in progress while status is RUNNING, QUEUED, RESUMING_WAREHOUSE, or BLOCKED, and it has finished when status is SUCCESS, FAILED_WITH_ERROR, or FAILED_WITH_INCIDENT:

[
  {
    "execution_id": "01b1f2e0-0000-df4f-0000-00100006589e",
    "status": "SUCCESS",
    "start_time": "2026-07-14T10:50:00Z",
    "end_time": "2026-07-14T10:50:39Z",
    "database_name": "MY_DB",
    "schema_name": "MY_SCHEMA",
    "warehouse": "MY_WAREHOUSE"
  }
]

Cancel a job¶

Cancel a running job by its job ID, using SQL or the REST API.

Using SQL:

Cancel the run’s query by ID with SYSTEM$CANCEL_QUERY:

SELECT SYSTEM$CANCEL_QUERY('<job_id>');

Using the REST API:

POST /api/v2/code-bundle-executions/{executionId}:cancel

View logs in the event table¶

Application logs are written to your account’s event table. Each record is tagged with the job’s query ID in RESOURCE_ATTRIBUTES['snow.query.id']. Find the event table, then query it by the job ID (replace <job_id> with the ID returned at submission):

SHOW PARAMETERS LIKE 'EVENT_TABLE' IN ACCOUNT;

SELECT TIMESTAMP, RECORD_TYPE, VALUE
FROM my_event_table
WHERE RESOURCE_ATTRIBUTES['snow.query.id'] = '<job_id>'
  AND RECORD_TYPE = 'LOG'
ORDER BY TIMESTAMP;

Drop the RECORD_TYPE filter to also see the METRIC and SPAN records emitted for the run.

View a failed job’s stack trace¶

To find why a job failed, query the same event table and filter to ERROR and FATAL severity. The results include the full stack trace, with source line numbers:

SELECT
  TIMESTAMP,
  RECORD['severity_text']::string AS severity,
  SCOPE['name']::string AS scope,
  VALUE::string AS body
FROM my_event_table
WHERE RESOURCE_ATTRIBUTES['snow.query.id'] = '<job_id>'
  AND RECORD_TYPE = 'LOG'
  AND RECORD['severity_text']::string IN ('ERROR', 'FATAL')
ORDER BY TIMESTAMP;

Snowflake also records structured exception attributes on these records, which you can select individually: RECORD_ATTRIBUTES['exception.type'], RECORD_ATTRIBUTES['exception.message'], and RECORD_ATTRIBUTES['exception.stacktrace'].

Spark Monitoring UI¶

You can browse your Spark runs in the Spark Monitoring UI in Snowsight. Sign in to Snowsight, then open the Spark run history at https://app.snowflake.com/<organization>/<account>/#/compute/history/spark (replace <organization> and <account> with your own). The page lists Spark runs over a selectable time range, such as the last 7 days. Each run appears as a single entry identified by the same job ID returned at submission, so you can set the time range and locate a run by its ID.