Using third-party packages¶

Get started¶

Use Snowflake’s default Artifact Repository (snowflake.snowpark.pypi_shared_repository) to connect and install PyPI packages within Snowpark UDFs and procedures.

Before you use this repository, the account administrator (a user who has been granted the ACCOUNTADMIN role) must grant the SNOWFLAKE.PYPI_REPOSITORY_USER database role to your role:

GRANT DATABASE ROLE SNOWFLAKE.PYPI_REPOSITORY_USER TO ROLE some_user_role;

The account administrator may also grant this database role to all users in the account:

GRANT DATABASE ROLE SNOWFLAKE.PYPI_REPOSITORY_USER TO ROLE PUBLIC;

With this role, you can install the package from the repository. When you create the UDF, you set the ARTIFACT_REPOSITORY parameter to the artifact repository name. You also set the PACKAGES parameter to the list of the names of the packages that will come from artifact repository. In the following example, because the artifact repository is configured with PyPI, the package scikit-learn is sourced from PyPI:

CREATE OR REPLACE FUNCTION sklearn_udf()
  RETURNS FLOAT
  LANGUAGE PYTHON
  RUNTIME_VERSION = 3.12
  ARTIFACT_REPOSITORY = snowflake.snowpark.pypi_shared_repository
  PACKAGES = ('scikit-learn')
  HANDLER = 'udf'
  AS
$$
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

def udf():
  X, y = load_iris(return_X_y=True)
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

  model = RandomForestClassifier()
  model.fit(X_train, y_train)
  return model.score(X_test, y_test)
$$;

SELECT sklearn_udf();

PACKAGES = ('scikit-learn==1.5')

Packages built only for x86¶

If a package is built only for x86, choose one of the warehouses that uses x86 CPU architecture — MEMORY_1X_x86 or MEMORY_16X_x86 — and then specify RESOURCE_CONSTRAINT=(architecture='x86'), as in the following example:

CREATE OR REPLACE FUNCTION pymeos_example()
RETURNS STRING
LANGUAGE PYTHON
HANDLER='main'
RUNTIME_VERSION='3.11'
ARTIFACT_REPOSITORY=snowflake.snowpark.pypi_shared_repository
PACKAGES=('pymeos') -- dependency pymeos-cffi is x86 only
RESOURCE_CONSTRAINT=(architecture='x86')
AS $$
def main() -> str:
   from pymeos import pymeos_initialize, pymeos_finalize, TGeogPointInst, TGeogPointSeq

   # Always initialize MEOS library
   pymeos_initialize()

   sequence_from_string = TGeogPointSeq(
      string='[Point(10.0 10.0)@2019-09-01 00:00:00+01, Point(20.0 20.0)@2019-09-02 00:00:00+01, Point(10.0 10.0)@2019-09-03 00:00:00+01]')

   sequence_from_points = TGeogPointSeq(instant_list=[TGeogPointInst(string='Point(10.0 10.0)@2019-09-01 00:00:00+01'),
        TGeogPointInst(string='Point(20.0 20.0)@2019-09-02 00:00:00+01'),
        TGeogPointInst(string='Point(10.0 10.0)@2019-09-03 00:00:00+01')],
          lower_inc=True, upper_inc=True)
   speed = sequence_from_points.speed()

   # Call finish at the end of your code
   pymeos_finalize()

   return speed
$$;

SELECT pymeos_example();

For more information, see Snowpark-optimized warehouses.

You can use Artifact Repository with UDF and Stored Procedure client APIs such as the following:

• snowflake.snowpark.udf.UDFRegistration

• snowflake.snowpark.functions.sproc

When using them, specify the following parameters:

• ARTIFACT_REPOSITORY

• PACKAGES

and provide the package name in the PACKAGES field.

See the following example:

...
ARTIFACT_REPOSITORY="snowflake.snowpark.pypi_shared_repository",
PACKAGES=["urllib3", "requests"],
...

Copy

Troubleshooting¶

If the package install fails for the function or procedure creation part, run the following pip command locally to see whether the package specification is valid:

pip install <package name> --only-binary=:all: --python-version 3.12 –platform <platform_tag>

Limitations¶

• Access to private repositories is not supported.

• You cannot use this feature directly in Notebooks. However, you can use a UDF or stored procedure that uses PyPI packages within a notebook.

• You cannot use Artifact Repository within anonymous stored procedures.

Licensing terms¶

• In Snowflake: Governed by your existing Snowflake customer agreement, including the Anaconda usage restrictions described in this documentation. No separate Anaconda terms apply for in-Snowflake use.

• Local development: From Snowflake’s dedicated Anaconda repository : Subject to Anaconda’s Embedded End Customer Terms and Anaconda’s Terms of Service posted on the repository. Local use is limited to developing/testing workloads intended for deployment in Snowflake.

User guidelines¶

Finding and managing packages¶

Can’t find a package you need?

• Submit requests via the Snowflake Ideas forum.

• Pure Python packages (without compiled extensions) can be uploaded directly to a Snowflake stage.

Support and security¶

Displaying and using packages¶

select * from information_schema.packages where language = 'python';

select * from information_schema.packages where (package_name = 'numpy' and language = 'python');

desc function stock_sale_average(varchar, number, number);

Performance on cold warehouses¶

For more efficient resource management, newly provisioned virtual warehouses do not preinstall Anaconda packages. Instead, Anaconda packages are installed on-demand the first time a UDF is used. The packages are cached for future UDF execution on the same warehouse. The cache is dropped when the warehouse is suspended. This may result in slower performance the first time a UDF is used or after the warehouse is resumed. The additional latency could be approximately 30 seconds.

Performance with single row prediction¶

Some data science frameworks, such as Scikit-learn and TensorFlow, might be slow when doing single-row ML prediction. To improve performance, do batch prediction instead of single-row prediction. To do this, you can use vectorized Python UDFs, with which you can define Python functions that receive input rows in batches, on which machine learning or data science libraries are optimized to operate. For more information, see Vectorized Python UDFs.

Downloading data on demand from data science libraries¶

Some data science libraries, such as NLTK, Keras, and spaCy provide functionality to download additional corpora, data, or models on demand.

However, on-demand downloading does not work with Python UDFs due to Snowflake security constraints, which disable some capabilities, such as network access and writing to files.

To work around this issue, download the data to your local environment and then provide it to the UDF via a Snowflake stage.

XGBoost¶

When using XGBoost in UDF or UDTF for parallel prediction or training, the concurrency for each XGBoost instance should be set to 1. This ensures that XGBoost is configured for optimal performance when executing in the Snowflake environment.

Examples:

import xgboost as xgb
model = xgb.Booster()
model.set_param('nthread', 1)
model.load_model(...)

import xgboost as xgb
model = xgb.XGBRegressor(n_jobs=1)

TensorFlow/Keras¶

When using Tensorflow/Keras for prediction, use Model.predict_on_batch and not Model.predict.

Example:

import keras
model = keras.models.load_model(...)
model.predict_on_batch(np.array([input]))