Monitor dbt Projects on Snowflake

This topic explains the ways you can use monitoring features for dbt Projects on Snowflake to inspect dbt project executions—–manual or task-scheduled–—and how to view logs and artifacts.

Section

Description

Enable monitoring features for dbt project objects

Capture logging and tracing events for a dbt project object and for any scheduled task that runs it. To enable this feature, you must set logging, tracing, and metrics on the schema where the dbt project object and task are deployed.

Monitor scheduled executions of dbt project objects

In Snowsight, in the navigation menu, select Transformation » dbt Projects to view run history, task graphs, and query details for dbt project objects. When a workspace is connected to a dbt project object that runs according to a task schedule, you can open task-run history and task graphs from within the workspace.

Access dbt artifacts and logs programmatically

Use the DBT_PROJECT_EXECUTION_HISTORY table function and dbt system functions to access dbt artifacts and logs programmatically.

Enable monitoring features for dbt project objects

To enable monitoring features for your dbt project object, set LOG_LEVEL, TRACE_LEVEL, and METRIC_LEVEL on the database and schema where your dbt project object is created, as shown in the following SQL example:

ALTER SCHEMA my_db.my_dbt_project_schema SET LOG_LEVEL = 'INFO';
ALTER SCHEMA my_db.my_dbt_project_schema SET TRACE_LEVEL = 'ALWAYS';
ALTER SCHEMA my_db.my_dbt_project_schema SET METRIC_LEVEL = 'ALL';

Monitor scheduled executions of dbt project objects

If you execute a deployed dbt project object on a schedule using a task, and the task is in the same schema as the dbt project object, you can view scheduled tasks directly from the workspace by selecting Connect and then View Schedules.

Note

This feature is only available for workspaces that are connected to a dbt project object.

To monitor scheduled execution of a dbt project object from a workspace:

  1. From the dbt project menu on the right side of the project pane, under Scheduled runs, choose View schedules.

  2. From the list, select the schedule (task) that you want to inspect, and then choose View details.

    The information pane for the task opens, where you can view Task details, the task Graph (if applicable), and Run History of this task. For more information, see View tasks and task graphs in Snowsight.

  3. From the Run History for any scheduled dbt project object execution in the list, select the Open query history button on the far right to view query details, the query profile, and the query telemetry for the run. For more information, see Review details and profile of a specific query.

Monitor dbt project objects in Snowsight

To view detailed monitoring information about dbt project object executions, navigate to Transformations » dbt Projects in Snowsight. You must use a role with the MONITOR privilege to view monitoring information for the dbt project object. For more information, see Access control for dbt projects on Snowflake.

  1. In the navigation menu, select Transformation » dbt Projects. A histogram shows the frequency of dbt project object executions and a list of projects that have run.

    The list of dbt project objects includes columns with the following information. You can filter the list by date range, command, and run status.

    • PROJECT - The name of the dbt project object and the number of executions (runs) in the selected time period.

    • LAST COMMAND - The dbt command that executed during the last run.

    • LAST RUN STATUS - The result of the run: Succeeded, Executing, or Failed.

    • LAST RUN - The elapsed time since the last run. To reverse the sort order, select the column header. The most recent run is shown first by default.

    • PREVIOUS RUNS - The number of runs in the selected time period by status.

    • DATABASE and SCHEMA - The database and schema where the dbt project object is saved.

    • LAST RUN PARAMETERS - The dbt command-line arguments (ARGS) specified in the EXECUTE DBT PROJECT command for the last dbt project object execution.

  2. To inspect individual project runs, select a dbt project object from the list.

    The dbt project object details page in the database object explorer opens for that dbt project object.

    The Run History tab is selected by default, with the following information for each job run in the selected time period:

    • COMMAND - The dbt command that executed during the last run.

    • STATUS - The result of the run: Succeeded, Executing, or Failed.

    • RUN TIME - The elapsed time since the last run. To reverse the sort order, select the column header. The most recent run is shown first by default.

    • PARAMETERS The dbt command-line arguments (ARGS) specified in the EXECUTE DBT PROJECT command for the last dbt project object execution.

  3. To see details for a run, select it from the list.

    The Query Details page opens for the EXECUTE DBT PROJECT query that ran. This page includes the following tabs:

    • Query Details - Displays the execution status, start time, end time, duration, warehouse size, query ID, and the SQL text of the EXECUTE DBT PROJECT command. Also shows per-model results for the dbt command that ran (for example, build), including each model name, time taken, and status.

    • Query Profile - Shows the query execution plan and performance statistics.

    • Query Telemetry - Shows telemetry data for the execution.

    • DAG - Visualizes the models that ran during the execution and their results. For more information, see View the query history DAG.

    For more information, see Review details and profile of a specific query.

View the query history DAG

The Query Details for a dbt project object execution includes a DAG tab that visualizes what ran during an execution and the results for each model. This differs from the DAG on the project details page, which serves as a documentation layer for your project, including models, tests, sources, and their dependencies.

The query history DAG is built from the manifest.json and run_results.json artifacts produced during an execution. Select a node in the DAG to open a side panel with details for that specific query, including the query ID and any error messages if the query failed.

To view the query history DAG:

  1. In the navigation menu, select Transformations » dbt Projects.

  2. From the list of dbt project objects, select a project.

  3. On the Run History tab, select a run to open the Query Details for that execution.

  4. Select the DAG tab.

Note

If the query history DAG shows “No data available,” the run likely failed before run_results.json could be generated. For more information, see Limitations for the query history DAG.

Access dbt artifacts and logs programmatically

Use the DBT_PROJECT_EXECUTION_HISTORY table function and the following system functions to access dbt artifacts and logs programmatically.

Function

What it returns

Typical use

Notes

SYSTEM$GET_DBT_LOG

Text log output (the run’s log tail)

Quick debugging in SQL. For example, see errors and warnings without downloading files.

Returns log content; nothing is created or moved.

SYSTEM$LOCATE_DBT_ARTIFACTS

Folder path (for example, snow://…/results/query_id_…/) containing artifact files such as manifest.json, compiled SQL, logs.

Browse or copy specific files with LIST, GET, or COPY FILES.

Just a locator (a URL); you still run GET/COPY FILES to fetch.

SYSTEM$LOCATE_DBT_ARCHIVE

Single ZIP file URL (for example, …/dbt_artifacts.zip).

Handy when you want to download one file (for example, with GET).

Use GET '<url>' file:///local/dir to download.

Get logs and download a ZIP file of the latest dbt project query

The following example queries Snowflake’s dbt execution history to show the most recent query ID for the dbt project object. It pulls the log output for that execution and returns the location of the zipped dbt artifacts for that execution.

The Snowflake CLI example downloads the artifacts ZIP file or specific files (like manifest.json) to your local folder using GET.

To download the ZIP file from Snowsight, navigate to Transformations » dbt Projects, select your project, then select an execution to navigate to Query Details, and select Download Build Artifacts under dbt Output.

You must use a role with the OWNERSHIP, USAGE, or MONITOR privilege on your dbt project objects.

Tip

Use function arguments such as DATABASE, SCHEMA, and OBJECT_NAME to filter results whenever possible. These filters are applied before the RESULT_LIMIT (default: 100 rows), so using them ensures you get the most relevant results rather than filtering a potentially truncated result set with a WHERE clause.

--Look up the most recent dbt project object execution
SET latest_query_id = (SELECT query_id
   FROM TABLE(SNOWFLAKE.INFORMATION_SCHEMA.DBT_PROJECT_EXECUTION_HISTORY(
     DATABASE => 'ANALYTICS_DB',
     SCHEMA => 'DBT_PROD',
     OBJECT_NAME => 'FINANCE_ANALYTICS'
   ))
   ORDER BY query_end_time DESC LIMIT 1);

--Get the dbt run logs for the most recent dbt project object execution
SELECT SYSTEM$GET_DBT_LOG($latest_query_id);

============================== 15:14:53.100781 | 46d19186-61b8-4442-8339-53c771083f16 ==============================
[0m15:14:53.100781 [info ] [Dummy-1   ]: Running with dbt=1.9.4
...
[0m15:14:58.198545 [debug] [Dummy-1   ]: Command `cli run` succeeded at 15:14:58.198121 after 5.19 seconds

To view the stage path where Snowflake stored the dbt project object execution artifacts, use the SYSTEM$LOCATE_DBT_ARTIFACTS function. You can then use that path with GET or COPY FILES with the Snowflake CLI to download things like manifest.json, compiled SQL, or logs.

--Get the location of the dbt project object archive ZIP file (see all files)
SELECT SYSTEM$LOCATE_DBT_ARTIFACTS($latest_query_id);

+-------------------------------------------------------------------------------------------------+
| SYSTEM$LOCATE_DBT_ARTIFACTS($LATEST_QUERY_ID)                                                   |
+-------------------------------------------------------------------------------------------------+
| snow://dbt/TESTDBT.PUBLIC.MY_DBT_PROJECT/results/query_id_01c01096-010c-0ccb-0000-a99506bd199e/ |
+-------------------------------------------------------------------------------------------------+

--List all the files of a dbt run
ls 'snow://dbt/TESTDBT.PUBLIC.MY_DBT_PROJECT/results/query_id_01bf3f5a-010b-4d87-0000-53493abb7cce/';

You can also create a fresh internal stage, locate the Snowflake-managed path for the specified dbt project object execution’s artifacts, and copy those artifacts into your stage for retrieval, as shown in the following example:

CREATE OR REPLACE STAGE my_dbt_stage ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');

SELECT SYSTEM$LOCATE_DBT_ARTIFACTS($latest_query_id);

snow://dbt/TESTDBT.PUBLIC.MY_DBT_PROJECT/results/query_id_01bf51c1-010b-5676-0000-53493ae6db02/

COPY FILES INTO @my_dbt_stage/results/ FROM 'snow://dbt/TESTDBT.PUBLIC.MY_DBT_PROJECT/results/query_id_01bf51c1-010b-5676-0000-53493ae6db02/';

results/dbt_artifacts.zip
results/logs/dbt.log
results/target/manifest.json
results/target/semantic_manifest.json