Snowpark ML: End-to-End Machine Learning in Snowflake¶

Snowpark ML Modeling¶

Snowpark ML Modeling supports data preprocessing, feature engineering, and model training in Snowflake using popular machine learning frameworks, such as scikit-learn, xgboost, and lightgbm. This API also includes a preprocessing module that can use compute resources provided by a Snowpark-optimized warehouse to provide scalable data transformations.

Snowpark ML Operations¶

Snowpark ML Operations (MLOps), featuring the Snowpark ML Model Registry, complements the Snowpark ML Development API. The model registry allows secure deployment and management of models in Snowflake, and supports models trained both inside and outside of Snowflake.

Snowpark ML Data Access¶

Snowpark ML Data Access provides simple and performant ways to feed data into your machine learning models.

• The FileSet API provides a Python fsspec-compliant API for materializing data into a Snowflake internal stage from a query or Snowpark DataFrame. It also provides convenient methods for working with the data and feeding it to PyTorch or TensorFlow.

• A set of framework connectors provide optimized, secure, and performant data provisioning for Pytorch and Tensorflow frameworks in their native data loader formats.

Installing Snowpark ML from the Snowflake conda Channel¶

1. Create the conda environment where you will install Snowpark ML. If you prefer to use an existing environment, skip this step.

   conda create --name snowpark-ml
   

   Copy

2. Activate the conda environment:

   conda activate snowpark-ml
   

   Copy

3. Install Snowpark ML from the Snowflake conda channel:

   conda install --override-channels --channel https://repo.anaconda.com/pkgs/snowflake/ snowflake-ml-python
   

   Copy

Installing Snowpark ML from PyPI¶

You can install the Snowpark ML package from the Python Package Index (PyPI) by using the standard Python package manager, pip.

1. Change to your project directory and activate your Python virtual environment:

   cd ~/projects/ml
   source .venv/bin/activate
   

   Copy

2. Install the Snowpark ML package:

   python -m pip install snowflake-ml-python
   

   Copy

Installing Optional Modeling Dependencies¶

Some Snowpark ML Modeling APIs require dependencies that are not installed as dependencies of Snowpark ML. The scikit-learn and xgboost packages installed by default when you install Snowpark ML Modeling, but lightgbm is an optional dependency. If you plan to use classes in the snowflake.ml.modeling.lightgbm namespace, install lightgbm yourself.

Use the following commands to activate your conda environment and install lightgbm from the Snowflake conda channel.

conda activate snowpark-ml
conda install --override-channels --channel https://repo.anaconda.com/pkgs/snowflake/ lightgbm

Use the following commands to activate your virtual environment and install lightgbm using pip.

.venv/bin/activate
python -m pip install 'snowflake-ml-python[lightgbm]'

Snowflake might add additional optional dependencies to Snowpark ML from time to time. To install all optional dependencies using pip:

.venv/bin/activate
python -m pip install 'snowflake-ml-python[all]'

Setting Up Snowpark Python¶

Snowpark Python is a dependency of Snowpark ML and is installed automatically with Snowpark ML. If Snowpark Python is not set up on your system, you might need to perform additional configuration steps. See Setting Up Your Development Environment for Snowpark Python for Snowpark Python setup instructions.

Specifying a Warehouse¶

Many parts of Snowpark ML, for example training a model or running inference, run code in a Snowflake warehouse. These operations run in the warehouse specified by the session you use to connect. For example, if you create a session from a named connection in your SnowSQL configuration file, you can specify a warehouse using the warehousename parameter in the named configuration.

You can add the warehouse setting when creating the Session object, as shown here, if it does not already exist in the configuration.

from snowflake.snowpark import Session
from snowflake.ml.utils import connection_params
# Get named connection from SnowQSL configuration file
params = connection_params.SnowflakeLoginOptions("myaccount")
# Add warehouse name for model method calls if it's not already present
if "warehouse" not in params:
    params["warehouse"] = "mlwarehouse"
sp_session = Session.builder.configs(params).create()

If no warehouse is specified in the session, or if you want to use a different warehouse, call the session’s use_warehouse method.

sp_session.use_warehouse("mlwarehouse")