Snowpark ML Modeling: ML Model Development¶

Snowpark ML Modeling Classes¶

All Snowpark ML modeling and preprocessing classes are in the snowflake.ml.modeling namespace. The Snowpark ML modules have the same name as the corresponding modules from the sklearn namespace. For example, the Snowpark ML module corresponding to sklearn.calibration is snowflake.ml.modeling.calibration.

The xgboost and lightgbm modules correspond to snowflake.ml.modeling.xgboost and snowflake.ml.modeling.lightgbm, respectively.

Not all of the classes from scikit-learn are supported in Snowpark ML. The following table indicates which classes are supported. Classes marked with an asterisk (^*) support distributed execution.

General API Differences¶

Snowpark ML Modeling includes data preprocessing, transformation, and prediction algorithms based on scikit-learn, xgboost, and lightgbm. The Snowpark Python classes are replacements for the corresponding classes from the original packages, with similar signatures. However, these APIs are designed to work with Snowpark DataFrames instead of NumPy arrays.

Although the Snowpark ML Modeling API is similar to scikit-learn, there are some key differences. This section explains how to call the __init__ (constructor), fit, and predict methods for the estimator and transformer classes provided in Snowpark ML.

• The constructor of all Snowpark ML Python classes accepts five additional parameters—input_cols, output_cols, sample_weight_col, label_cols, and drop_input_cols—in addition to the parameters accepted by the equivalent classes in scikit-learn, xgboost, or lightgbm. These are strings or sequences of strings that specify the names of the input columns, output columns, sample weight column, and label columns in a Snowpark or Pandas DataFrame.

• The fit and predict methods in Snowpark ML accept a single DataFrame instead of separate arrays representing input data, labels, and weights. With Snowpark ML, you specify the names of the columns to be used for these purposes when you instantiate the class; these names are then used to find the required columns in the DataFrame that you pass to fit or predict. See fit and predict.

• The transform and predict methods in Snowpark ML return a DataFrame containing all of the columns from the DataFrame passed to the method, with the output from the prediction stored in additional columns. (You can transform in place by specifying the same input and output column names or drop the input columns by passing drop_input_cols = True.) The scikit-learn, xgboost, and lightgbm equivalents return arrays containing only the results.

• Snowpark Python transformers do not have a fit_transform method. However, as with scikit-learn, parameter validation is only performed in the fit method, so you should call fit at some point before transform, even when the transformer does not do any fitting. fit returns the transformer, so the method calls may be chained; for example, Binarizer(threshold=0.5).fit(df).transform(df).

• Snowpark ML transformers do not have an inverse_transform method. This method is unnecessary with Snowpark ML because the original representation remains available in the input columns of the input DataFrame, which are preserved unless you explicitly perform an in-place transform by specifying the same names for both the input and output columns.

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier()

model.fit(
    X=df[feature_column_names], y=df[label_column_names], sample_weight=df[weight_column_name]
)

model.fit(df)

prediction_results = model.predict(X=df[feature_column_names])

prediction_results = model.predict(df)[output_column_names]

Deploying and Running Your Model¶

The result of training a model is a Python Snowpark ML model object. You can use the trained model to make predictions by calling the model’s predict method. This creates a temporary user-defined function to run the model in your Snowflake virtual warehouse. This function is automatically deleted at the end of your Snowpark ML session (for example, when your script ends or when you close your Jupyter notebook).

To keep the user-defined function after your session ends, you can create it manually. See the Quickstart on the topic for further information.

The Snowpark ML model registry, an upcoming feature, also supports persistent models and makes finding and deploying them easier. For early access to documentation on this feature, contact your Snowflake representative.

Pipeline for Multiple Transformations¶

With scikit-learn, it is common to run a series of transformations using a pipeline. scikit-learn pipelines do not work with Snowpark ML classes, so Snowpark ML provides a Snowpark Python version of sklearn.pipeline.Pipeline for running a series of transformations. This class is in the snowflake.ml.modeling.pipeline package, and it works the same as the scikit-learn version.

Known Limitations¶

• Snowpark ML estimators and transformers do not currently support sparse inputs or sparse responses. If you have sparse data, convert it to a dense format before passing it to Snowpark ML’s estimators or transformers.

• The Snowpark ML package does not currently support matrix data types. Any operation on estimators and transformers that would produce a matrix as a result fails.

• The order of rows in result data is not guaranteed to match the order of rows in input data.

Troubleshooting¶

import logging, sys

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)

Further Reading¶

See the documentation of the original libraries for complete information on their functionality.

• Scikit-Learn

• XGBoost

• LightGBM

Acknowledgement¶

Some parts of this document are derived from the Scikit-learn documentation, which is licensed under the BSD-3 “New” or “Revised” license and Copyright © 2007-2023 The scikit-learn developers. All rights reserved.

Some parts of this document are derived from the XGboost documentation, which is covered by Apache License 2.0, January 2004 and Copyright © 2019. All rights reserved.

Some parts of this document are derived from the LightGBM documentation, which is MIT-licensed and Copyright © Microsoft Corp. All rights reserved.