Using Third-Party Packages¶

Getting Started¶

Before you start using the packages provided by Anaconda inside Snowflake, you must acknowledge the Snowflake Third Party Terms.

1. Sign in to Snowsight.

2. Select Admin » Billing & Terms.

3. In the Anaconda section, select Enable.

4. In the Anaconda Packages dialog, click the link to review the Snowflake Third Party Terms page.

5. If you agree to the terms, select Acknowledge & Continue.

If you see an error when attempting to accept the terms of service, your user profile might be missing a first name, last name, or email address. If you have an administrator role, refer to Setting User Details and Preferences to update your profile using Snowsight. Otherwise, contact an administrator to update your account.

Displaying and Using Packages¶

select * from information_schema.packages where language = 'python';

select * from information_schema.packages where (package_name = 'numpy' and language = 'python');

desc function stock_sale_average(varchar, number, number);

Performance On Cold Warehouses¶

For more efficient resource management, newly provisioned virtual warehouses do not preinstall Anaconda packages. Instead, Anaconda packages are installed on-demand the first time a UDF is used. The packages are cached for future UDF execution on the same warehouse. The cache is dropped when the warehouse is suspended. This may result in slower performance the first time a UDF is used or after the warehouse is resumed. The additional latency could be approximately 30 seconds.

Performance With Single Row Prediction¶

Some data science frameworks, such as Scikit-learn and TensorFlow, might be slow when doing single-row ML prediction.

To improve performance, do batch prediction instead of single-row prediction. To do this, convert the input dataset into Snowflake arrays and feed the arrays into the prediction code. The batching can be done with tabular Python UDTFs, the ARRAY_AGG function, Tabular Java UDFs (UDTFs), or Tabular JavaScript UDFs (UDTFs).

You can also use the vectorized Python UDFs, with which you can define Python functions that receive input rows in batches, on which machine learning or data science libraries are optimized to operate. For more information, see Vectorized Python UDFs.

Downloading Data On Demand From Data Science Libraries¶

Some data science libraries, such as NLTK, Keras, and spaCy provide functionality to download additional corpora, data, or models on demand.

However, on-demand downloading does not work with Python UDFs due to Snowflake security constraints, which disable some capabilities, such as network access and writing to files.

To work around this issue, download the data to your local environment and then provide it to the UDF via a Snowflake stage.

XGBoost¶

When using XGBoost in UDF or UDTF for parallel prediction or training, the concurrency for each XGBoost instance should be set to 1. This ensures that XGBoost is configured for optimal performance when executing in the Snowflake environment.

Examples:

import xgboost as xgb
model = xgb.Booster()
model.set_param('nthread', 1)
model.load_model(...)

import xgboost as xgb
model = xgb.XGBRegressor(n_jobs=1)

TensorFlow/Keras¶

When using Tensorflow/Keras for prediction, use Model.predict_on_batch and not Model.predict.

Example:

import keras
model = keras.models.load_model(...)
model.predict_on_batch(np.array([input]))