Provisioned Throughput¶

Overview¶

Use Provisioned Throughput to reserve throughput for managed inference on Snowflake Cortex. You specify throughput size as provisioned throughput units (PTU), and Cortex allocates the required capacity for a one-month term. You can use the PTUs in your REST API calls for a consistent end-user experience. The functionality is available for the following models in the AWS and Azure Clouds:

• Mistral Large 2

• Llama 3.1-405B

• Llama 3.1-70B

• Llama 3.1-8B

• Snowflake-Llama3.3-70B

• Snowflake-Llama3.3-405B

Access control requirements¶

Users must use a role that has been granted the SNOWFLAKE.CORTEX_USER database role with USAGE privilege on the PT ID. For more information about this privilege, see Privileges.

GRANT CREATE PROVISIONED THROUGHPUT ON ACCOUNT TO ROLE <role>

Minimum Provisioned Throughput Unit requirements¶

Provisioned Throughput is subject to minimum and incremental PTU requirements. Each model or feature in the Minimum PTUs column shows the minimum number of PTUs that you must request. If you request fewer PTUs than the minimum, your request is rejected.

If you need more throughput than the minimum PTUs offer for the model, you need additional PTUs. The Increment PTUs column shows the PTU increments in excess of the Minimum PTUs that you can request. Requests must specify PTUs such that the amount exceeding the minimum is a whole integer multiple of the increment; otherwise, the request is rejected.

The table below lists the available models, the minimum PTUs required for each model, and the increment requirements for additional PTUs beyond the minimum.

Determining PTU size¶

The PTUs required for your application depend on the workload profile. For example, on Llama 3.1-8B, a workload with 500 requests per minute (RPM) and 500 tokens per request output has a minimum of 64 PTUs. It delivers 960K tokens of throughput per minute. If you need more throughput, you can request additional PTUs in increments of 32.

When you’re starting out, you can use the minimum PTUs for the model and add increments as needed.

Cost considerations¶

For the duration of your Provisioned Throughput term, you consume Credits per PTU per hour at the rate listed in the Snowflake Credit Consumption Table. You incur charges for the allocated PTUs regardless of your actual usage during the term. The term starts and ends at 8:00 a.m. PT for the dates provided in the provisioned throughput creation.

Provisioned Throughput does not renew automatically. To reserve throughput for another term, see the following section.

Reserving throughput¶

This tutorial guides you through the process of reserving and using provisioned throughput in a REST API call for the Cortex COMPLETE function.

CREATE PROVISIONED THROUGHPUT my_pt CLOUD_PROVIDER='aws', MODEL='llama3.1-8B', PTUS=64, TERM_START='2025-04-15' TERM_END='2025-05-15'

DESCRIBE PROVISIONED THROUGHPUT my_pt

curl --location 'https://some-account-identifier.snowflakecomputing.com/api/v2/cortex/inference:complete' \
--header 'X-Snowflake-Authorization-Token-Type: KEYPAIR_JWT' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: ••••••' \
--data '{
  "model": "snowflake-llama-3.1-8b",
  "messages": [
  {
      "content": "Write an essay on the benefits of provisioned throughput."
  }
  ],
  "provisioned_throughput_id": "f3a27d60-f61f-4247-8aa3-6272ea0d7a8d"
}'