Using third-party packages¶

Get started¶

Snowflake has a default Artifact Repository called snowflake.snowpark.pypi_shared_repository that you use to connect and install PyPI packages within Snowpark UDFs and procedures. For a user to use this repository, the account administrator (a user who has been granted the ACCOUNTADMIN role) needs to grant the SNOWFLAKE.PYPI_REPOSITORY_USER database role to the user’s role:

GRANT DATABASE ROLE SNOWFLAKE.PYPI_REPOSITORY_USER TO ROLE some_user_role;

To grant this role to all users in the account, the account administrator can grant the SNOWFLAKE.PYPI_REPOSITORY_USER database role to the PUBLIC role:

GRANT DATABASE ROLE SNOWFLAKE.PYPI_REPOSITORY_USER TO ROLE PUBLIC;

Then, you install the package from the repository. When you create the UDF, you set the ARTIFACT_REPOSITORY parameter to the artifact repository name. You also set the PACKAGES parameter to the list of the name of the packages that would come from artifact repository. In the below example, since artifact repository is configured with PyPI, the package scikit-learn is sourced from PyPI.

CREATE OR REPLACE FUNCTION sklearn_udf()
  RETURNS FLOAT
  LANGUAGE PYTHON
  RUNTIME_VERSION = 3.9
  ARTIFACT_REPOSITORY = snowflake.snowpark.pypi_shared_repository
  ARTIFACT_REPOSITORY_PACKAGES = ('scikit-learn')
  HANDLER = 'udf'
  AS
$$
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

def udf():
  X, y = load_iris(return_X_y=True)
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

  model = RandomForestClassifier()
  model.fit(X_train, y_train)
  return model.score(X_test, y_test)
$$;

SELECT sklearn_udf();

Packages built only for x86¶

If a package is built only for x86, choose one of the warehouses that uses x86 CPU architecture: MEMORY_1X_x86, MEMORY_16X_x86. For more information, see Snowpark-optimized warehouses.

Then specify: RESOURCE_CONSTRAINT=(architecture='x86'). See an example below.

CREATE OR REPLACE PROCEDURE TEST_PRESIDIO(x)
  RETURNS STRING
  LANGUAGE PYTHON
  RESOURCE_CONSTRAINT=(architecture='x86')
  RUNTIME_VERSION = 3.9
  HANDLER = 'main'
  ARTIFACT_REPOSITORY = snowflake.snowpark.pypi_shared_repository
  ARTIFACT_REPOSITORY_PACKAGES = ('presidio_analyzer','presidio_anonymizer')
  AS
  $$
from presidio_analyzer import AnalyzerEngine

# Set up the engine, loads the NLP module (spaCy model by default) and other PII recognizers
analyzer = AnalyzerEngine()

# Call analyzer to get results
results = analyzer.analyze(text="My phone number is 425-xxx-xxxx",
entities=["PHONE_NUMBER"],
language='en')
return results
$$;

You can use Artifact Repository with UDF and Stored Procedure client APIs such as the following:

• snowflake.snowpark.udf.UDFRegistration

• snowflake.snowpark.functions.sproc

When using them, specify the following parameters:

• ARTIFACT_REPOSITORY

• ARTIFACT_REPOSITORY_PACKAGES

See the example below:

...
ARTIFACT_REPOSITORY="snowflake.snowpark.pypi_shared_repository",
ARTIFACT_REPOSITORY_PACKAGES=["urllib3", "requests"],
...

Copy

Troubleshooting¶

Package install fails for the function or procedure creation part. You can run the following pip command locally to see if the package specification is valid:

pip install <package name> --only-binary=:all: --python-version 3.9 –platform <platform_tag>

Limitations¶

• Currently, Snowflake allows only binary distributions (wheel files).

• Access to private repositories is not supported.

• You cannot use this feature directly in Notebooks. You can however, use a UDF or stored procedure that uses PyPI packages within a notebook.

• You cannot use Artifact Repo within anonymous stored procedures.

Getting started¶

Before you start using the packages provided by Anaconda inside Snowflake, you must acknowledge the External Offerings Terms.

1. Sign in to Snowsight.

2. Select Admin » Terms.

3. In the Anaconda section, select Enable.

4. In the Anaconda Packages dialog, click the link to review the External Offerings Terms page.

5. If you agree to the terms, select Acknowledge & Continue.

If you encounter an error when attempting to accept the External Offerings Terms, it may be due to missing information in your user profile, such as a first name, last name, or email address. If you have administrator privileges, see Add user details to your user profile to update your profile using Snowsight. Otherwise, contact an administrator to update your account.

Displaying and using packages¶

select * from information_schema.packages where language = 'python';

select * from information_schema.packages where (package_name = 'numpy' and language = 'python');

desc function stock_sale_average(varchar, number, number);

Performance on cold warehouses¶

For more efficient resource management, newly provisioned virtual warehouses do not preinstall Anaconda packages. Instead, Anaconda packages are installed on-demand the first time a UDF is used. The packages are cached for future UDF execution on the same warehouse. The cache is dropped when the warehouse is suspended. This may result in slower performance the first time a UDF is used or after the warehouse is resumed. The additional latency could be approximately 30 seconds.

Performance with single row prediction¶

Some data science frameworks, such as Scikit-learn and TensorFlow, might be slow when doing single-row ML prediction. To improve performance, do batch prediction instead of single-row prediction. To do this, you can use vectorized Python UDFs, with which you can define Python functions that receive input rows in batches, on which machine learning or data science libraries are optimized to operate. For more information, see Vectorized Python UDFs.

Downloading data on demand from data science libraries¶

Some data science libraries, such as NLTK, Keras, and spaCy provide functionality to download additional corpora, data, or models on demand.

However, on-demand downloading does not work with Python UDFs due to Snowflake security constraints, which disable some capabilities, such as network access and writing to files.

To work around this issue, download the data to your local environment and then provide it to the UDF via a Snowflake stage.

XGBoost¶

When using XGBoost in UDF or UDTF for parallel prediction or training, the concurrency for each XGBoost instance should be set to 1. This ensures that XGBoost is configured for optimal performance when executing in the Snowflake environment.

Examples:

import xgboost as xgb
model = xgb.Booster()
model.set_param('nthread', 1)
model.load_model(...)

import xgboost as xgb
model = xgb.XGBRegressor(n_jobs=1)

TensorFlow/Keras¶

When using Tensorflow/Keras for prediction, use Model.predict_on_batch and not Model.predict.

Example:

import keras
model = keras.models.load_model(...)
model.predict_on_batch(np.array([input]))