Model Serving in Snowpark Container Services¶

Create a compute pool¶

Snowpark Container Services (SPCS) runs container images in compute pools. If you don’t already have a suitable compute pool, create one as follows:

CREATE COMPUTE POOL IF NOT EXISTS mypool
    MIN_NODES = 2
    MAX_NODES = 4
    INSTANCE_FAMILY = 'CPU_X64_M'
    AUTO_RESUME = TRUE;

See the family names table for a list of valid instance families.

Make sure the role that will run the model is the owner of the compute pool or else has the USAGE or OPERATE privilege on the pool.

Create an image repository¶

Snowflake Model Serving builds a container image that holds the model and its dependencies. To store this image, you need an image repository. If you don’t already have one, create one as follows:

CREATE IMAGE REPOSITORY IF NOT EXISTS my_inference_images

If you will be using an image repository that you do not own, make sure the role that will build the container image has the READ, WRITE, SERVICE READ, and SERVICE WRITE privileges on the repository. Grant these privileges as follows:

GRANT READ ON IMAGE REPOSITORY my_inference_images TO ROLE myrole;
GRANT WRITE ON IMAGE REPOSITORY my_inference_images TO ROLE myrole;
GRANT SERVICE READ ON IMAGE REPOSITORY my_inference_images TO ROLE myrole;
GRANT SERVICE WRITE ON IMAGE REPOSITORY my_inference_images TO ROLE myrole;

Model dependencies and eligibility¶

A model’s dependencies determine if it can run in a warehouse, in a SPCS service, or both. You can, if necessary, intentionally specify dependencies to make a model ineligible to run in one of these environments.

The Snowflake conda channel is available only in warehouses and is the only source for warehouse dependencies. By default, conda dependencies for SPCS models obtain their dependencies from conda-forge.

When you log a model version, the model’s conda dependencies are validated against the Snowflake conda channel. If all the model’s conda dependencies are available there, the model is deemed eligible to run in a warehouse. It may also be eligible to run in a SPCS service if all of its dependencies are available from conda-forge, although this is not checked until you create a service.

Models logged with PyPI dependencies must be run on SPCS. Specifying at least one PyPI dependency is one way to make a model ineligible to run in a warehouse. If your model has only conda dependencies, specify at least one with an explicit channel (even conda-forge), as shown in the following example.

reg.log_model(
    model_name="my_model",
    version_name="v1",
    model=model,
    conda_dependencies=["conda-forge::scikit-learn"])

For SPCS-deployed models, conda dependencies, if any, are installed first, then any PyPI dependencies are installed in the conda environment using pip.

Create a service¶

To create a SPCS service and deploy the model to it, call the model version’s create_service method, as shown in the following example.

mv.create_service(service_name="myservice",
                  service_compute_pool="my_compute_pool",
                  image_repo="mydb.myschema.my_image_repo",
                  ingress_enabled=True,
                  gpu_requests=None)

The following the required arguments to create_service:

• service_name: The name of the service to create. This name must be unique within the account.

• service_compute_pool: The name of the compute pool to use to run the model. The compute pool must already exist.

• image_repo: The name of the image repository to use to store the container image. The repo must already exist and the user must have the SERVICE WRITE privilege on it (or OWNERSHIP).

• ingress_enabled: If True, the service is made accessible via an HTTP endpoint. To create the endpoint, the user must have the BIND SERVICE ENDPOINT privilege.

• gpu_requests: A string specifying the number of GPUs. For a model that can be run on either CPU or GPU, this argument determines whether the model will be run on the CPU or on the GPUs. If the model is of a known type that can only be run on CPU (for example, scikit-learn models), the image build fails if GPUs are requested.

This example shows only the required and most commonly used arguments. See the ModelVersion API reference for a complete list of arguments.

Default service configuration¶

By default, a CPU-powered model uses twice the number of CPUs, plus one, worker processes. GPU-powered models use one worker process. You can override this using the num_workers argument.

Some models are not thread-safe. Therefore, the service loads a separate copy of the model for each worker process. This can result in resource depletion for large models.

By default, the inference server optimizes for running a single inference at a time, aiming to make full use of all CPU and memory of each node.

Container image build behavior¶

By default, Snowflake Model Serving builds the container image using the same compute pool that will be used to run the model. This inference compute pool is likely overpowered for this task (for example, GPUs are not used in building container images). In most cases, this won’t have a significant impact on compute costs, but if it is a concern, you can choose a less powerful compute pool for building images by specifying the image_build_compute_pool argument.

create_service is an idempotent function. Calling it multiple times does not trigger image building every time. However, container images may be rebuilt based on updates in the inference service, including fixes for vulnerabilities in dependent packages. When this happens, create_service automatically triggers a rebuild of the image.

SQL¶

Snowflake Model Serving creates service functions when deploying a model to SPCS. These functions serve as a bridge from SQL to the model running in the SPCS compute pool. One service function is created for each method of the model, and they are named like model_name_method_name. For example, if the model has two methods named PREDICT and EXPLAIN and is being deployed to a service named MY_SERVICE, the resulting service functions are MY_SERVICE_PREDICT and MY_SERVICE_EXPLAIN.

Calling a model’s service functions in SQL is done using code like the following:

SELECT MY_SERVICE_PREDICT(...) FROM ...;

Python¶

Call a service’s methods using the run method of a model version object, including the service_name argument to specify the service where the method will run. For example:

service_prediction = mv.run(
    test_df,
    function_name="predict",
    service_name="my_service")

If you do not include the service_name argument, the model runs in a warehouse.

HTTP endpoint¶

After deploying a service with ingress enabled, a new HTTP endpoint is available for the service. You can find the endpoint using the ShOW ENDPOINTS IN SERVICE command.

SHOW ENDPOINTS IN SERVICE my_service;

Take note of the ingress_url column, which should look like random_str-account-id.snowflakecomputing.app.

To learn more about using this endpoint, see the SPCS tutorial Create a Snowpark Container Services service and the topic Using a service from outside Snowflake in the Developer Guide. For more information about the required data format, see Remote service input and output data formats.

See Deploying a Hugging Face sentence transformer for GPU-powered inference for an example of using a model service HTTP endpoint.

Suspending services¶

Once you are done using a model, or if there is no usage, it is a good practice to suspend the service to save costs. You can do this using the ALTER SERVICE command.

ALTER SERVICE my_service SUSPEND;

The service restarts automatically when it receives a request, subject to scheduling and startup delays. Scheduling delay depends on the availability of the compute pool, and startup delay depends on the size of the model.

Deleting models¶

You can manage models and model versions as usual with either the SQL interface or the Python API, with the restriction that a model or model version that is being used by a service (whether running or suspended) cannot be dropped (deleted). To drop a model or model version, drop the service first.

Deploying an XGBoost model for CPU-powered inference¶

THe following code illustrates the key steps in deploying an XGBoost model for inference in SPCS, then using the deployed model for inference. A notebook for this example is available.

from snowflake.ml.registry import registry
from snowflake.ml.utils.connection_params import SnowflakeLoginOptions
from snowflake.snowpark import Session
from snowflake.ml.modeling.xgboost import XGBRegressor

# your model training code here output of which is a trained xgb_model

# Open model registry
reg = registry.Registry(session=session, database_name='my_registry_db', schema_name='my_registry_schema')

# Log the model in Snowflake Model Registry
model_ref = reg.log_model(
    model_name="my_xgb_forecasting_model",
    version_name="v1",
    model=xgb_model,
    conda_dependencies=["scikit-learn","xgboost"],
    sample_input_data=train,
    comment="XGBoost model for forecasting customer demand"
)

# Deploy the model to SPCS
reg_model.create_service(
    service_name="ForecastModelServicev1",
    service_compute_pool="my_cpu_pool",
    image_repo="my_db.data.my_images",
    ingress_enabled=True)

# See all services running a model
reg_model.list_services()

# Run on SPCS
reg_model.run(input_data, function_name="predict", service_name="ForecastModelServicev1")

# Delete the service
reg_model.delete_service("ForecastModelServicev1")

Deploying a Hugging Face sentence transformer for GPU-powered inference¶

This code trains and deploys a Hugging Face sentence transformer, including an HTTP endpoint.

from snowflake.ml.registry import registry
from snowflake.ml.utils.connection_params import SnowflakeLoginOptions
from snowflake.snowpark import Session
from sentence_transformers import SentenceTransformer

session = Session.builder.configs(SnowflakeLoginOptions("connection_name")).create()
reg = registry.Registry(session=session, database_name='my_registry_db', schema_name='my_registry_schema')

# Take an example sentence transformer from HF
embed_model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')

# Have some sample input data
input_data = [
    "This is the first sentence.",
    "Here's another sentence for testing.",
    "The quick brown fox jumps over the lazy dog.",
    "I love coding and programming.",
    "Machine learning is an exciting field.",
    "Python is a popular programming language.",
    "I enjoy working with data.",
    "Deep learning models are powerful.",
    "Natural language processing is fascinating.",
    "I want to improve my NLP skills.",
]

# Log the model with pip dependencies
pip_model = reg.log_model(
    embed_model,
    model_name="sentence_transformer_minilm",
    version_name='pip',
    sample_input_data=input_data,  # Needed for determining signature of the model
pip_requirements=["sentence-transformers", "torch", "transformers"], # If you want to run this model in the Warehouse, you can use conda_dependencies instead
)

# Force Snowflake to not try to check warehouse
conda_forge_model = reg.log_model(
    embed_model,
    model_name="sentence_transformer_minilm",
    version_name='conda_forge_force',
    sample_input_data=input_data,
    # setting any package from conda-forge is sufficient to know that it can't be run in warehouse
    conda_dependencies=["sentence-transformers", "conda-forge::pytorch", "transformers"]
)

# Deploy the model to SPCS
pip_model.create_service(
    service_name="my_minilm_service",
    service_compute_pool="my_gpu_pool",  # Using GPU_NV_S - smallest GPU node that can run the model
    image_repo="my_db.data.my_images",
    ingress_enabled=True,
    gpu_requests="1", # Model fits in GPU memory; only needed for GPU pool
    max_instances=4, # 4 instances were able to run 10M inferences from an XS warehouse
)

# See all services running a model
pip_model.list_services()

# Run on SPCS
pip_model.run(input_data, function_name="encode", service_name="my_minilm_service")

# Delete the service
pip_model.delete_service("my_minilm_service")

Since this model has ingress enabled, you can call its HTTP endpoint as follows.

import json
from pprint import pprint
import requests
import snowflake.connector

# Generate right header
# Note that, ideally user should use key-pair authentication for API access (see this).
def initiate_snowflake_connection():
    connection_parameters = SnowflakeLoginOptions("connection_name")
    connection_parameters["session_parameters"] = {"PYTHON_CONNECTOR_QUERY_RESULT_FORMAT": "json"}
    snowflake_conn = snowflake.connector.connect(**connection_parameters)
    return snowflake_conn

def get_headers(snowflake_conn):
    token = snowflake_conn._rest._token_request('ISSUE')
    headers = {'Authorization': f'Snowflake Token=\"{token["data"]["sessionToken"]}\"'}
    return headers

headers = get_headers(initiate_snowflake_connection())

# Put the endpoint url with method name
URL='https://<random_str>-<account>.snowflakecomputing.app/encode'

# Prepare data to be sent
data = {
    'data': []
}
for idx, x in enumerate(input_data):
    data['data'].append([idx, x])

# Send over HTTP
def send_request(data: dict):
    output = requests.post(URL, json=data, headers=headers)
    assert (output.status_code == 200), f"Failed to get response from the service. Status code: {output.status_code}"
    return output.content

# Test
results = send_request(data=data)
pprint(json.loads(results))

Deploying a PyTorch model for GPU-powered inference¶

See this quickstart for an example of training and deploying a PyTorch deep learning recommendation model (DLRM) to SPCS for GPU inference.

Monitoring SPCS Deployments¶

You can monitor deployment by inspecting the services being launched using the following SQL query.

SHOW SERVICES IN COMPUTE POOL my_compute_pool;

Two jobs are launched:

• MODEL_BUILD_xxxxx: The final characters of the name are randomized to avoid name conflicts. This job builds the image and ends after the image has been built. If an image already exists, the job is skipped.

  The logs are useful for debugging issues such as conflicts in package dependencies. To see the logs from this job, run the SQL below, being sure to use the same final characters:

  CALL SYSTEM$GET_SERVICE_LOGS('MODEL_BUILD_xxxxx', 0, 'model-build');
  

  Copy

• MYSERVICE: The name of the service as specified in the call to create_service. This job is started if the MODEL_BUILD job is successful or skipped. To see the logs from this job, run the SQL below:

  CALL SYSTEM$GET_SERVICE_LOGS('MYSERVICE', 0, 'model-inference');
  

  Copy

Package conflicts¶

Two systems dictate the packages installed in the service container: the model itself and the inference server. To minmize conflicts with your model’s dependencies, the inference server requires only the following packages:

• gunicorn<24.0.0

• starlette<1.0.0

• uvicorn-standard<1.0.0

Make sure your model dependencies, along with the above, are resolvable by pip or conda, whichever you use.

If a model has both conda_dependencies and pip_requirements set, these will be installed as follows via conda:

Channels:

• conda-forge

• nodefaults

Dependencies:

• all_conda_packages

• pip:

  • all_pip_packages

By default, Snowflake uses conda-forge for Anaconda packages, since the Snowflake conda channel is available in warehouses, and the defaults channel requires users to accept Anaconda terms of use. To specify packages from the defaults channel, include the package name: defaults::pkg_name.

Service out of memory¶

Some models are not thread-safe, so Snowflake loads a copy of the model in memory for each worker process. This can cause out-of-memory conditions for large models with a higher number of workers. Try reducing num_workers.

Unsatisfactory query performance¶

Usually, inference is bottlenecked by the number of instances in the inference service. Try passing a higher value for max_instances while deploying the model.

Unable to alter the service spec¶

The specifications of the model build and inference services cannot be changed using ALTER SERVICE. You can only change attributes such as TAG, MIN_INSTANCES, and so forth.

Since the image is published in the image repo, however, you can copy the spec, modify it, and create a new service from it, which you can start manually.