Monitor and troubleshoot DCM Projects

This topic describes how to monitor DCM deployments and troubleshoot failing DCM plans.

Troubleshoot a DCM project

If you are unfamiliar with the DCM project, you might run into errors from misconfigurations or other common pitfalls. This section describes those errors and how to resolve them.

Common causes for errors

The following table lists common causes for errors in a DCM project execution:

Error category

Common causes

Secondary roles

  • Users encounter inconsistent behavior due to unknowingly leveraging secondary role privileges when running DCM commands.

Insufficient role privileges

  • Insufficient role privileges to create defined object types

  • Insufficient role privileges to alter or drop existing objects that are now owned by another role

  • Insufficient role privileges to use system-DMFs

  • Insufficient role privileges to run a warehouse to refresh a dynamic table at creation

Jinja rendering issues

  • Jinja rendering issues from Incorrect Jinja syntax

  • Jinja rendering issues from Value-type mismatches

Project issues

  • Incorrect manifest path

  • Empty definition folders

  • Outdated definition files on the wrong repo branch

  • Objects that are already deployed by another DCM project

  • Mismatched project and object references

Observe and audit DCM project deployments

DCM Projects are designed to provide full transparency and audit trails for all changes to your account infrastructure. This requires you to follow a few software development best practices for setting up infrastructure deployment processes. For more information, see Automate a DCM project deployment.

Use the following sources to review previous deployments:

Deployment artifacts

For every executed deployment, an immutable snapshot of the deployment artifacts is stored inside the DCM project, with the following information:

  • The manifest file (manifest.yml)

  • All object definition and macro files (.sql files) inside the sources folder

  • The output of the PLAN operation (plan_result.json) and the DEPLOY operation (deploy_result.json), including:

    • The templating variables used for this deployment

    • Deployment metadata, including timestamp, object name, and query ID

    • The changeset

This complete set makes all deployment actions reproducible for debugging, auditing, or redeploying the defined state.

The following commands are available for observing and auditing a DCM project:

  • With the MONITOR privilege, you can:

    • List all deployments stored inside the DCM project.

    • List all files inside a specified deployment.

    • Read, copy, or download specific files inside that deployment.

  • With the OWNERSHIP privilege, you can manually drop a deployment if it contains sensitive data.

  • With the READ privilege, you can run the DESCRIBE command to see the most recent deployment name, alias, and timestamp for a selected DCM project.

Example commands:

DESCRIBE DCM PROJECT DCM_DEMO.PROJECTS.DCM_PROJECT_DEV;

SHOW DEPLOYMENTS IN DCM PROJECT DCM_DEMO.PROJECTS.DCM_PROJECT_DEV;

LIST 'snow://project/DCM_DEMO.PROJECTS.DCM_PROJECT_DEV/deployments/DEPLOYMENT$1/';

ALTER DCM PROJECT DCM_DEMO.PROJECTS.DCM_PROJECT_DEV DROP DEPLOYMENT DEPLOYMENT$1;

Deployment history

The INFORMATION_SCHEMA functions provide role-based access and low-latency ways to see successful and failed deployments for a selected DCM project.

The arguments project_name and result_limit are optional.

Example commands:

SELECT
  START_TIMESTAMP,
  END_TIMESTAMP,
  STATUS,
  DEPLOYMENT_NUMBER,
  DEPLOYMENT_ALIAS,
  CONFIGURATION_PROFILE,
  ERROR_MESSAGE,
  EXECUTOR_ROLE,
  STATS
FROM
  TABLE (DCM_DEMO.INFORMATION_SCHEMA.DCM_DEPLOYMENT_HISTORY(
    project_name => 'DCM_DEMO.PROJECTS.DCM_PROJECT_DEV',
    result_limit => 50
  ));

Event logs

You can set the preferred LOG_LEVEL on the DCM project object or inherit the defined LOG_LEVEL from the parent schema, database, or account.

If the LOG_LEVEL for the DCM project is set, failed PLAN and DEPLOY executions are logged with the corresponding error messages as an event, and you can see them by querying the defined event table. For more information about setting up event tables and log levels, see Event table overview.

For example:

SELECT
  TIMESTAMP,
  RESOURCE_ATTRIBUTES:"snow.executable.name" ::STRING AS PROJECT_NAME,
  CASE
    WHEN RESOURCE_ATTRIBUTES:"snow.project.dcm.execution.command" ::STRING = 'plan' THEN 'PLAN'
    WHEN RESOURCE_ATTRIBUTES:"snow.project.dcm.execution.command" ::STRING = 'deploy' THEN 'DEPLOY'
    ELSE RESOURCE_ATTRIBUTES:"snow.project.dcm.execution.command" ::STRING
  END AS COMMAND,
  CASE
    WHEN VALUE:"state" ::STRING = 'SUCCEEDED' THEN 'SUCCEEDED'
    WHEN VALUE:"state" ::STRING = 'FAILED' THEN 'FAILED'
    ELSE VALUE:"state" ::STRING
  END AS STATUS,
  COALESCE(
    CONCAT('Error message: ',VALUE:"message"::STRING),
    VALUE:"operation"::STRING)
  AS OPERATIONS,
  RESOURCE_ATTRIBUTES:"snow.session.role.primary.name" ::STRING AS ROLE,
  RESOURCE_ATTRIBUTES:"db.user" ::STRING AS USER_NAME,
  RECORD:"severity_text" ::STRING AS SEVERITY
FROM
  SNOWFLAKE.TELEMETRY.EVENTS
WHERE
  RESOURCE_ATTRIBUTES:"snow.executable.type" ::STRING = 'DCM_PROJECT'
ORDER BY
  TIMESTAMP DESC
LIMIT
  250;