Using third-party packages¶

Importing packages through a Snowflake stage¶

Snowflake stages can be used to import packages. You can bring in any Python code that follows guidelines defined in General limitations. For more information, see Creating a Python UDF with code uploaded from a stage.

You can only upload pure Python packages or packages with native code through a Snowflake stage. If your uploaded package has dependency on x86 CPU architecture, then you must use Snowpark-optimized warehouses and use the RESOURCE_CONSTRAINT warehouse property, with CPU architecture x86.

As an example, you can use the following SQL, which creates a warehouse named so_warehouse that has x86 CPU architecture:

CREATE WAREHOUSE so_warehouse WITH
   WAREHOUSE_SIZE = 'LARGE'
   WAREHOUSE_TYPE = 'SNOWPARK-OPTIMIZED'
   RESOURCE_CONSTRAINT = 'MEMORY_16X_X86';

To install a package with native code via importing from Stage, use the following example:

create or replace function native_module_test_zip()
returns string
language python
runtime_version=3.9
handler='compute'
imports=('@mystage/mycustompackage.zip')
as
$$
import os
import sys
def compute():
   from zipfile import ZipFile
   if os.path.exists('/tmp/mycustompackage'):
      shutil.rmtree('/tmp/mycustompackage')
   os.mkdir('/tmp/mycustompackage')
   zippath = os.path.join(sys._xoptions["snowflake_import_directory"], 'mycustompackage.zip')
   ZipFile(zippath).extractall('/tmp/mycustompackage')
   sys.path.insert(0, '/tmp/mycustompackage')
   #return os.listdir('/tmp/mycustompackage')
   import mycustompackage
   return mycustompackage.mycustompackage()
$$;

Using third-party packages from Anaconda¶

For convenience, a number of popular open source third-party Python packages that are built and provided by Anaconda are made available to use out of the box inside Snowflake virtual warehouses. There is no additional cost for such use of the Anaconda packages apart from Snowflake’s standard consumption-based pricing. With the exception of Snowflake Notebooks, Anaconda packages are currently not eligible for use within Snowpark Container Services (SPCS). To use Python packages in a container image for SPCS, you can install these packages from PyPi using pip.

To view the list of third-party packages from Anaconda, see the Anaconda Snowflake channel.

To request the addition of new packages, go to the Snowflake Ideas page in the Snowflake Community. Select the Python Packages & Libraries category and check if someone has already submitted a request. If so, vote on it. Otherwise, click New Idea and submit your suggestion.

select * from information_schema.packages where language = 'python';

select * from information_schema.packages where (package_name = 'numpy' and language = 'python');

desc function stock_sale_average(varchar, number, number);

Local development and testing¶

To help you create a conda environment on your local machine for development and testing, Anaconda has created a Snowflake channel which mirrors a subset of the packages and versions that are supported in the Snowflake Python UDF environment. You may use the Snowflake conda channel for local testing and development at no cost under the Supplemental Embedded Software Terms to Anaconda’s Terms of Service.

For example, to create a new conda environment locally using the Snowflake channel, type something like this on the command line:

conda create --name py38_env -c https://repo.anaconda.com/pkgs/snowflake python=3.9 numpy pandas

Note that because of platform differences, your local conda environment may not be exactly the same as the server environment.

Best practices¶

Within the create function statement, the package specification (for example, packages = ('numpy','pandas')) should only specify the top-level packages that the UDF is using directly. Anaconda performs dependency management of packages and will install the required dependencies automatically. Because of this, you should not specify dependency packages.

Anaconda will install the most up-to-date version of the package and its dependencies if you don’t specify a package version. Generally, it isn’t necessary to specify a particular package version. Note that version resolution is performed once, when the UDF is created using the create function command. After that, the resulting version resolution is frozen and the same set of packages will be used when this particular UDF executes.

For an example of how to use the package specification within the create function statement, see Importing a package in an in-line handler.

Known issues with third-party packages¶

import xgboost as xgb
model = xgb.Booster()
model.set_param('nthread', 1)
model.load_model(...)

import xgboost as xgb
model = xgb.XGBRegressor(n_jobs=1)

import keras
model = keras.models.load_model(...)
model.predict_on_batch(np.array([input]))