Run Python Jobs in Snowflake¶

This topic walks you through developing, deploying, and scheduling Python scripts in Snowflake on Warehouse or Compute Pools. You can write and test your code locally using your favorite IDE or with Cortex Code, then deploy it as a Notebook Project Object — a schema-levelobject that runs your script non-interactively on a warehouse or compute pool. Once deployed, you can run your project on demand or schedule it with a task.

Prerequisites¶

To complete this guide, you need the following on your development machine:

• Python 3.10, 3.11, or 3.12

Set up your project¶

1. Clone the sample Python project:

   Copy codeExpand code block

   git clone https://github.com/sfc-gh-jfreeberg/npo.git

   Show lessSee more

   Scroll to top

2. Create a virtual environment and install the dependencies:

   Copy codeExpand code block

   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r requirements-prod.txt -r requirements-dev.txt

   Show lessSee more

   Scroll to top

The template project contains the following files:

• src/main.py — the main entry point for your script
• src/args_example.py — shows how to pass and process command-line arguments
• src/udf_example.py — shows how to register and reference UDFs
• requirements-prod.txt — production dependencies (installed at run time in Snowflake)
• requirements-dev.txt — development dependencies, including the Snowflake CLI
• snow_app.yml — configuration for runtime, compute, and dependencies
• tests/ — a sample test suite

Develop and test locally¶

Run the demo script locally to verify your environment:

python src/main.py

Run the test suite:

python -m pytest

You can extend the sample scripts to read or write data specific to your team or use case. Iterate locally until your script is ready, then configure and deploy it to Snowflake.

Deploy to Snowflake¶

Deploy your project to Snowflake so it can be run non-interactively.

1. Create the Notebook Project Object:
   Copy codeExpand code block

   snow notebook project create my_project --source . --overwrite
   # You can change the project name, "my_project" to your desired name

   Show lessSee more

   Scroll to top

The project is now deployed to your Snowflake account.

Execute your Project¶

Run your project using the Snowflake CLI:

snow notebook project execute -n 'my_project' --main-file 'src/main.py'

You can run any script in the project by specifying the main file.

snow notebook project execute -n 'my_project' --main-file 'src/args_example.py'

snow notebook project execute -n 'my_project' --main-file 'src/udf_example.py'

You can also run the project using SQL:

EXECUTE NOTEBOOK PROJECT my_db.my_schema.my_project
  MAIN_FILE = 'src/main.py';

Schedule Your Project¶

To run your project on a recurring schedule, wrap the run command in a Snowflake Task:

CREATE OR REPLACE TASK my_task
  WAREHOUSE = my_warehouse
  SCHEDULE = 'USING CRON 0 0 * * * UTC'
AS
  EXECUTE NOTEBOOK PROJECT my_db.my_schema.my_project
    MAIN_FILE = 'src/ain.py';

After creating the task, resume it to start the schedule:

ALTER TASK my_task RESUME;

Automate deployment with CI/CD¶

You can run the deploy commands in a CI/CD pipeline to automatically deploy your project when you merge changes. The following example shows a GitHub Actions workflow that deploys on push to the main branch:

name: Deploy to Snowflake
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: Snowflake-Labs/snowflake-cli-action@v1
        with:
          cli-version: "latest"
          default-config-file-path: "config.toml"

      - run: snow notebook project create my_project --source . --overwrite

Adapt this pattern to your CI system by running snow notebook project create --source . --overwrite after authenticating the Snowflake CLI.

Register UDFs from your project¶

If your script depends on User Defined Functions and you want to manage their lifecycle through the project, create them during the script’s run with session.udf.register or the @udf decorator. See src/udf_example.py in the sample project for an example.

Reference¶

The snow_app.yml config¶

Before you deploy, configure runtime and dependency settings in the snow_app.yml file at the root of your project. This file sets default values so you don’t have to specify them every time you run the project.

An example of the snow_app.yml file is shown below:

# This configuration will run the project files on Python 3.12 on the active warehouse
# when the EXECUTE NOTEBOOK PROJECT command or `snow notebook project execute` command is used.
runtime:
  version: python_3.12

dependencies: requirements.txt

# This configuration will run the project files on the SYSTEM_COMPUTE_POOL_CPU compute pool
# and use my_warehouse for any queries from the script when the EXECUTE NOTEBOOK PROJECT command
# or `snow notebook project execute` command is used.
runtime:
  version: "V2.3-CPU-PY3.12"
  compute_pool: SYSTEM_COMPUTE_POOL_CPU
  query_warehouse: "my_warehouse"

runtime¶

Use the runtime property to specify the runtime version, compute platform, and other properties for running your project.

runtime.version¶

Required. The python runtime version to run the project on.

Allowed values for Warehouse:

• python_3.10
• python_3.11
• python_3.12

Allowed values for Compute Pools:

• V2.3-CPU-PY3.10
• V2.3-CPU-PY3.11
• V2.3-CPU-PY3.12
• V2.3-GPU-PY3.10
• V2.3-GPU-PY3.11
• V2.3-GPU-PY3.12

For the full list of allowed values, see EXECUTE NOTEBOOK PROJECT.

runtime.compute_pool¶

The compute pool to run the project on. If not specified, the project runs on the active warehouse of the session.

runtime.query_warehouse¶

The warehouse to use for SQL queries when a runtime.compute_pool is specified. If not specified, queries run on the active warehouse of the session.

dependencies¶

A reference to the requirements.txt file that lists the Python dependencies for the project. Dependencies are installed from the built-in snowflake.snowpark.pypi_shared_repository PyPI repository.

dependencies: "requirements.txt"

If you split production and development dependencies, specify your production dependencies:

dependencies: "requirements-prod.txt"

artifact_repository¶

The artifact repository to use for the project. If not specified, the project uses the built-in snowflake.snowpark.pypi_shared_repository PyPI repository.

artifact_repository: "db.schema.my_private_repo"

SQL commands¶

CREATE NOTEBOOK PROJECT¶

Create a project from a stage:

CREATE NOTEBOOK PROJECT [ IF NOT EXISTS ] <database_name>.<schema_name>.<project_name>
  FROM '@<stage_path>'
  [ COMMENT = '<string_literal>' ];

Create a project from a workspace:

CREATE NOTEBOOK PROJECT <database_name>.<schema_name>.<project_name>
  FROM 'snow://workspace/<workspace_path>'
  [ COMMENT = '<string_literal>' ];

For full syntax details, see CREATE NOTEBOOK PROJECT.

EXECUTE NOTEBOOK PROJECT¶

EXECUTE NOTEBOOK PROJECT <database_name>.<schema_name>.<project_name>
  MAIN_FILE = 'main.py'
  [ COMPUTE_POOL = '<compute_pool_name>' ]
  [ QUERY_WAREHOUSE = '<warehouse_name>' ]
  [ RUNTIME = '<runtime_version>' ]
  [ EXTERNAL_ACCESS_INTEGRATIONS = ( <integration_name> [ , ... ] ) ];

For full syntax details, see EXECUTE NOTEBOOK PROJECT.

Snowflake CLI commands¶

snow notebook project create¶

Usage: snow notebook project create [OPTIONS] [NAME]

Creates a notebook project in Snowflake.

╭─ Arguments ───────────────────────────────────────────────────────────────────────────────────────────────╮
│   name      [NAME]  Name of the Snowpark project.                                                         │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --source                  TEXT  Source location of the notebook project. Supports stage path (starting    │
│                                 with '@') or workspace path (starting with 'snow://workspace/').          │
│ --comment                 TEXT  Comment for the notebook project.                                         │
│ --overwrite                     Overwrite the notebook project if it already exists.                      │
│ --skip-if-exists                Skip the creation of the notebook project if it already exists.           │
│ --help            -h            Show this message and exit.                                               │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯

snow notebook project execute¶

Usage: snow notebook project execute [OPTIONS] [NAME] ARGUMENTS

Executes a notebook project in Snowflake.

╭─ Arguments ──────────────────────────────────────────────────────────────────────────────────────────────────╮
│   name           [NAME]  Name of the notebook project.                                                       │
│   arguments      TEXT    Arguments to pass to the notebook project.                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --main-file                             TEXT  Main file of the notebook project.                             │
│ --compute-pool                          TEXT  Compute pool to run the notebook project on.                   │
│ --query-warehouse                       TEXT  Query warehouse to run the notebook project on.                │
│ --runtime                               TEXT  Runtime to run the notebook project on.                        │
│ --requirements-file                     TEXT  Requirements file to use for the notebook project.             │
│ --external-access-integrations          TEXT  External access integrations to use for the notebook project.  │
│ --help                          -h            Show this message and exit.                                    │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

snow notebook project drop¶

Usage: snow notebook project drop [OPTIONS] [NAME]

Drops a notebook project in Snowflake.

╭─ Arguments ──────────────────────────────────────────────────────────────────────────────────────────────────╮
│   name      [NAME]  Name of the notebook project.                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help  -h        Show this message and exit.                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

snow notebook project list¶

Usage: snow notebook project list [OPTIONS]

Lists notebook projects in Snowflake.

╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --help  -h        Show this message and exit.                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Limitations¶

The following limitations apply:

• .ipynb notebooks can only run on compute pools, not warehouses. .py files can run on either.
• Java and Scala are not supported.
• Event table logging is not available when running on warehouses.
• Only requirements.txt is supported for specifying dependencies. pyproject.toml is not supported.