Introduction to tasks¶

Serverless tasks¶

The serverless compute model for tasks enables you to rely on compute resources managed by Snowflake instead of user-managed virtual warehouses. Snowflake automatically resizes serverless compute resources as required for each workload. Snowflake determines the ideal size of serverless compute resources for a given run based on a dynamic analysis of statistics for the most recent runs of the same task. The maximum compute size for a serverless task is equivalent to an XXLARGE user-managed virtual warehouse.

To use the serverless compute model, leave out the WAREHOUSE parameter when you create a task using CREATE TASK. Note that the role that executes the CREATE TASK command must have the global EXECUTE MANAGED TASK privilege. For more information about the access control requirements for tasks, see Task security.

Billing for runs of serverless tasks is different from the standard credit consumption model for tasks that rely on virtual warehouses for compute resources. For more information, see Task costs.

You can take some control over the cost and performance of serverless tasks by setting the following parameters:

• SERVERLESS_TASK_MAX_STATEMENT_SIZE: the maximum size of warehouse for running on serverless tasks to prevent unexpected costs.

• SERVERLESS_TASK_MIN_STATEMENT_SIZE: the minimum size of warehouse for predictable performance on serverless tasks.

• TARGET_COMPLETION_INTERVAL: the desired task completion time. Snowflake automatically resizes serverless compute resources to complete tasks within the time specified in the target completion interval. When this value is not set for scheduled tasks, Snowflake resizes serverless compute resources to complete before the next scheduled execution time.

  • If a series of task executions complete within the value specified on TARGET_COMPLETION_INTERVAL or a factor less, Snowflake scales down serverless compute to the next size so that subsequent task executions complete within the specified TARGET_COMPLETION_INTERVAL.

  • If a series of task executions are a factor greater than TARGET_COMPLETION_INTERVAL, Snowflake scales up serverless compute to the next size so that subsequent task executions complete within the specified TARGET_COMPLETION_INTERVAL.

When these three settings are specified, the precedence is as follows:

• SERVERLESS_TASK_MAX_STATEMENT_SIZE

• SERVERLESS_TASK_MIN_STATEMENT_SIZE

• TARGET_COMPLETION_INTERVAL

Here are some examples of using these parameters to control the serverless tasks.

Example 1: Create a serverless task to run every hour with a target completion interval of 120 minutes:

CREATE TASK SCHEDULED_T1 SCHEDULE='USING CRON 0 * * * * America/Los_Angeles'
TARGET_COMPLETION_INTERVAL='120 MINUTE' AS SELECT 1;

By setting TARGET_COMPLETION_INTERVAL with a relatively long duration, this indicates it’s acceptable for the task execution to run longer. Snowflake automatically sizes serverless compute so that the task execution completes within the specified TARGET_COMPLETION_INTERVAL.

Example 2: Create a serverless task to run every hour with a target completion time of 10 minutes:

CREATE TASK SCHEDULED_T2 SCHEDULE='USING CRON 0 * * * * UTC'
TARGET_COMPLETION_INTERVAL='10 MINUTE'
SERVERLESS_TASK_MAX_STATEMENT_SIZE='LARGE'
AS SELECT 1;

By setting TARGET_COMPLETION_INTERVAL with a relatively short duration, this indicates a preference for the task to quickly complete. Snowflake automatically sizes serverless compute so that the task execution completes within the specified TARGET_COMPLETION_INTERVAL until the SERVERLESS_TASK_MAX_STATEMENT_SIZE is reached.

Example 3: Create a serverless task scheduled once a day with a target completion interval and range of minimum and maximum warehouse sizes:

CREATE TASK SCHEDULED_T3 SCHEDULE='USING CRON 0 0 * * * UTC'
TARGET_COMPLETION_INTERVAL='180 M'
SERVERLESS_TASK_MIN_STATEMENT_SIZE='MEDIUM' SERVERLESS_TASK_MAX_STATEMENT_SIZE='LARGE' AS SELECT 1;

With this configuration, the serverless task runs once a day at midnight on at least a medium warehouse with a TARGET_COMPLETION_INTERVAL of 3 hours. Snowflake automatically sizes serverless compute so that the task execution completes within the specified TARGET_COMPLETION_INTERVAL until the SERVERLESS_TASK_MAX_STATEMENT_SIZE is reached.

Example 4: Create a serverless task scheduled once a day without a target completion interval:

CREATE TASK SCHEDULED_T4 SCHEDULE='USING CRON 0 0 * * * UTC'
SERVERLESS_TASK_MAX_STATEMENT_SIZE='LARGE' AS SELECT 1;

With this configuration, the serverless task runs once a day at midnight. If a series of task execution durations don’t complete before the next scheduled time, Snowflake automatically sizes serverless compute so that the task execution completes within the specified SCHEDULE until the SERVERLESS_TASK_MAX_STATEMENT_SIZE is reached.

For more information about the parameters and syntax, see CREATE TASK and ALTER TASK.

User-managed tasks¶

You can alternatively manage the compute resources for individual tasks by specifying an existing virtual warehouse when you create the task. This option requires that you choose a warehouse that is sized appropriately for the SQL actions that are executed by the task.

Choosing a warehouse size¶

If you choose to use existing warehouses to supply the compute resources for individual tasks, follow the best practices described in Warehouse considerations. To understand compute needs for your task, analyze the average run time for a single task or task graph using different warehouses based on warehouse size and clustering. You should also consider whether the warehouse is shared by multiple processes.

To analyze the average run time of your tasks, query the TASK_HISTORY account usage view. The average difference between the scheduled and completed times for a task is the expected average run time for the task. This difference includes the time the task was queued while other processes are using the compute resources in the warehouse.

For task graphs, there is a brief lag after a predecessor task finishes running and any child task starts. Choose a warehouse size large enough to accommodate multiple child tasks running simultaneously.

The following diagram shows a window of 1 minute in which a single task queued for 20 seconds and then ran for 40 seconds. This means that other processes were using the warehouse resources for the first 20 seconds after the task was scheduled.

The following diagram shows a task graph that requires 5 minutes on average to complete for each run. The diagram shows the window for 2 runs of the task graph to complete. This window is calculated from the time the root task is scheduled to start until the last child task has completed running.

In this example, the warehouse the task graph is running on is shared with other concurrent operations. These concurrent operations consume all available resources when each task in the task graph finishes running and before the next task starts running. As a result, the window for each task includes some amount of queuing while it waits for other operations to finish and free up compute resources.

Recommendations for choosing a compute model¶

The following table describes various factors that can help you decide when to use serverless tasks versus user-managed tasks:

The maximum size for a serverless task run is equivalent to an XXLARGE warehouse. If a task workload requires a larger warehouse, create a user-managed task with a warehouse of the required size.

Task scheduling¶

Tasks generally run on a schedule. You can define the schedule when creating a task using CREATE TASK or later using ALTER TASK.

A standalone task or the root task in a task graph generally runs on a schedule. You can define the schedule when creating a task using CREATE TASK or later using ALTER TASK.

Snowflake ensures only one instance of a task with a schedule is executed at a time. If a task is still running when the next scheduled execution time occurs, then that scheduled time is skipped.

Snowflake automatically resizes and scales the compute resources for serverless tasks. For user-managed tasks, choose an appropriate warehouse size for the task to complete its workload within the schedule. For information, see Choosing a Warehouse Size.

Triggered Tasks¶

You can use Triggered Tasks to run only when a defined stream has new data. This eliminates the need to poll a source frequently when the availability of new data is unpredictable. It also reduces latency because data is processed immediately.

Triggered Tasks don’t use compute resources until the defined stream has data and triggers the task run.

The following are run conditions for Triggered Tasks:

• When data is changed in the table that the associated stream tracks.

• When the task is first resumed, to consume any data already in the stream.

• Triggered Tasks automatically run a health check every 12 hours to prevent the stream from becoming stale. If no data is in the stream, Snowflake skips the run without using compute resources.

CREATE TASK triggeredTask  WAREHOUSE = my_warehouse
  WHEN system$stream_has_data('my_stream')
  AS
    INSERT INTO my_downstream_table
    SELECT * FROM my_stream;

ALTER TASK triggeredTask RESUME;

ALTER TASK task SUSPEND;
ALTER TASK task UNSET SCHEDULE;
ALTER TASK task RESUME;

Manually executing tasks¶

The EXECUTE TASK command manually triggers a single run of a task. This SQL command is useful for testing new or modified tasks before you enable them to execute SQL code in production.

You can call this SQL command directly in scripts or in stored procedures. In addition, this command supports integrating tasks in external data pipelines. Any third-party services that can authenticate into your Snowflake account and authorize SQL actions can execute the EXECUTE TASK command to run tasks.

Versioning of task runs¶

When a standalone task is first resumed or manually executed, an initial version of the task is set. The standalone task runs using this version. After a task is suspended and modified, a new version is set when the standalone task is resumed or manually executed.

When the task is suspended, all future scheduled runs of the task are cancelled; however, currently running tasks continue to run using the current version.

For example, suppose the task is suspended, but a scheduled run of this task has already started. The owner of the task modifies the SQL code called by the task while the task is still running. The task runs and executes the SQL code in its definition using the version of the task that was current when the task started its run. When the task is resumed or is manually executed, a new version of the task is set. This new version includes the modifications to the task.

To retrieve the history of task versions, query TASK_VERSIONS Account Usage view (in the SNOWFLAKE shared database).

Automatically suspend tasks after failed runs¶

Optionally suspend tasks automatically after a specified number of consecutive runs that either fail or time out. This feature can reduce costs by suspending tasks that consume Snowflake credits but fail to run to completion.

Set the SUSPEND_TASK_AFTER_NUM_FAILURES = num parameter on a task. When the parameter is set to a value greater than 0, tasks are automatically suspended after the specified number of consecutive task runs either fail or time out.

The parameter can be set when creating a task using CREATE TASK or later using ALTER TASK. You can also change this value in Snowsight.

The SUSPEND_TASK_AFTER_NUM_FAILURES parameter can also be set at the account, database, or schema level. The setting applies to all tasks contained in the modified object. Note that explicitly setting the parameter at a lower level overrides the parameter value set at a higher level.

Automatically retry failed task runs¶

If any task completes in a FAILED state, Snowflake can automatically retry the task. The automatic task retry is disabled by default. To enable this feature, set TASK_AUTO_RETRY_ATTEMPTS to a value greater than 0.

Tasks that use error notifications send notifications for each failed retry attempt. For more information, see Configuring a task to send error notifications.

When you set the TASK_AUTO_RETRY_ATTEMPTS parameter value at the account, database, or schema level, the change is applied to tasks contained in the modified object during their next scheduled run.

Access control privileges¶

Create custom roles to manage task permissions¶

With custom roles you can easily manage permissions granted to each account or role in Snowflake. To make changes to permissions for all accounts or roles using the custom role, update the custom role. Or, revoke permissions by removing the custom role.

USE SYSADMIN;

CREATE ROLE warehouse_task_creation
  COMMENT = 'This role can create user-managed tasks.';

USE ACCOUNTADMIN;

GRANT CREATE TASK
  ON SCHEMA schema1
  TO ROLE warehouse_task_creation;

GRANT USAGE
  ON WAREHOUSE warehouse1
  TO ROLE warehouse_task_creation;

USE SYSADMIN;

CREATE ROLE serverless_task_creation
  COMMENT = 'This role can create serverless tasks.';

USE ACCOUNTADMIN;

GRANT CREATE TASK
  ON SCHEMA schema1
  TO ROLE serverless_task_creation;

GRANT EXECUTE MANAGED TASK ON ACCOUNT
  TO ROLE serverless_task_creation;

USE ROLE securityadmin;

CREATE ROLE taskadmin;

USE ROLE accountadmin;

GRANT EXECUTE TASK, EXECUTE MANAGED TASK ON ACCOUNT TO ROLE taskadmin;

USE ROLE securityadmin;

GRANT ROLE taskadmin TO ROLE myrole;

Drop a task owner role¶

When you delete the owner role of a task, the task transfers ownership to the role that dropped the owner role. When a task transfers ownership, it is automatically paused and new executions aren’t scheduled until the new owner resumes the task.

If you drop the role while the task is running, the task run completes processing under the dropped role.

System service task execution¶

Snowflake runs tasks with the privileges of the task owner, but task runs are not associated with a user. Instead, each run is executed by a system service. Tasks are decoupled from specific users to avoid complications that can arise when users are dropped, locked due to authentication issues, or have roles removed.

Because task runs are decoupled from a user, the query history for task runs are associated with the system service. As such, there are no user credentials for this service, and no individual can assume its identity. Activity for the system service is limited to your account. The same encryption protections and other security protocols are built into this service as are enforced for other operations.