Snowflake DCM Projects — Extended capabilities (early access)¶

Introduction¶

This document describes a rolling set of new DCM Project capabilities that are available in private preview to selected customers. These features extend the core DCM Projects functionality with additional object types and deployment capabilities.

Over time, this document will be extended with new capabilities as they become available for early testing. Once a capability is sufficiently tested and stable, it will progress into the Public Preview release of DCM Projects and be removed from this document.

Early access for the following DCM capabilities is currently available in private preview:

• DEFINE FUNCTION (Python and Java)
• DEFINE PROCEDURE (Python and Java)
• DEFINE STREAMLIT
• DEFINE DBT PROJECT
• CLI enhancements
• DDL Hooks

DEFINE FUNCTION (Python and Java)¶

The publicly available DCM release supports only SQL-language user-defined functions. This early-access capability extends that support to Python and Java handlers.

Python handler¶

Inline the handler code in the AS $$...$$ block, exactly as you would in a CREATE FUNCTION statement:

DEFINE FUNCTION DEMO{{env_suffix}}.ANALYTICS.MULTIPLY(x float, y float)
    RETURNS FLOAT
    LANGUAGE PYTHON
    RUNTIME_VERSION = '3.11'
    HANDLER = 'udf.run'
AS $$
def run(x, y):
    return x * y
$$;

To reference a handler uploaded as a staged file, use the IMPORTS clause and omit the inline body:

DEFINE FUNCTION DEMO{{env_suffix}}.ANALYTICS.MY_FUNC(x float)
    RETURNS FLOAT
    LANGUAGE PYTHON
    RUNTIME_VERSION = '3.11'
    HANDLER = 'my_module.my_func'
    IMPORTS = ('@DEMO{{env_suffix}}.INGEST.MY_STAGE/my_module.py')
;

Java handler¶

DEFINE FUNCTION DEMO{{env_suffix}}.ANALYTICS.ADD_NUMBERS(x int, y int)
    RETURNS INT
    LANGUAGE JAVA
    RUNTIME_VERSION = '11'
    HANDLER = 'MathUtils.add'
AS $$
class MathUtils {
    public static int add(int x, int y) {
        return x + y;
    }
}
$$;

Functional limitations¶

• All CREATE OR ALTER FUNCTION limitations apply.
• PLAN only validates that the function can be created or altered successfully. It doesn’t check whether it will run successfully, as handler logic is compiled at runtime.
• PLAN treats a changed handler body as a full replace, the same way it handles SQL handlers. There’s no diff of the handler source code in the PLAN output.
• Inline handler bodies (AS $$...$$), staged imports (IMPORTS), and Artifactory references (ARTIFACT_REPOSITORY) are supported. External functions aren’t supported.
• Staged files referenced in IMPORTS must be uploaded to that Stage outside of DCM. DCM doesn’t support uploading files into user stages.

DEFINE PROCEDURE (Python and Java)¶

The publicly available DCM release supports only SQL-language stored procedures. This early-access capability extends that support to Python and Java handlers.

Python handler¶

DEFINE PROCEDURE DEMO{{env_suffix}}.ANALYTICS.PROCESS_DATA(input varchar)
    RETURNS VARCHAR
    LANGUAGE PYTHON
    RUNTIME_VERSION = '3.11'
    PACKAGES = ('snowflake-snowpark-python')
    HANDLER = 'sproc.run'
AS $$
def run(session, input):
    return input.upper()
$$;

To reference a handler uploaded as a staged file, use the IMPORTS clause and omit the inline body:

DEFINE PROCEDURE DEMO{{env_suffix}}.ANALYTICS.MY_PROC(input varchar)
    RETURNS VARCHAR
    LANGUAGE PYTHON
    RUNTIME_VERSION = '3.11'
    PACKAGES = ('snowflake-snowpark-python')
    HANDLER = 'my_module.run'
    IMPORTS = ('@DEMO{{env_suffix}}.INGEST.MY_STAGE/my_module.py')
;

Java handler¶

DEFINE PROCEDURE DEMO{{env_suffix}}.ANALYTICS.TRANSFORM(input varchar)
    RETURNS VARCHAR
    LANGUAGE JAVA
    RUNTIME_VERSION = '11'
    PACKAGES = ('com.snowflake:snowpark:latest')
    HANDLER = 'DataTransformer.run'
AS $$
import com.snowflake.snowpark_java.Session;

public class DataTransformer {
    public String run(Session session, String input) {
        return input.toUpperCase();
    }
}
$$;

Functional limitations¶

• All CREATE OR ALTER PROCEDURE limitations apply.
• PLAN only validates that the procedure can be created or altered successfully. It doesn’t check whether it will run successfully, as procedure logic is compiled at runtime.
• PLAN treats a changed handler body as a full replace, the same way it handles SQL handlers. There’s no diff of the handler source code in the PLAN output.
• Inline handler bodies (AS $$...$$) and staged imports (IMPORTS) are supported.
• Staged files referenced in IMPORTS must be uploaded to that Stage outside of DCM. DCM doesn’t support uploading files into user stages.

DEFINE STREAMLIT¶

You can define one or more Streamlit apps, their infrastructure, underlying tables, and access control together in a single DCM Project folder, then deploy everything to any environment with one command.

This is especially useful for dashboard and data app deployments that depend on objects (tables, views, dynamic tables) also managed by DCM. The entire stack (data pipeline and the app consuming it) can be version-controlled and promoted through environments together.

Create a DCM Project for Streamlit¶

You can take your existing Streamlit app folder, which includes:

• streamlit_app.py (or any entrypoint file)
• environment.yml (for warehouse runtime) or requirements.txt or pyproject.toml (for container runtime)
• Any supporting Python modules, pages, or asset files

and move it inside the sources/ folder of your DCM Project, since the Snowflake CLI only uploads files from the sources/ folder hierarchy. Place it outside of sources/definitions/ and organize it in any subfolder structure you like, for example sources/streamlit/my_dashboard/.

my_dcm_project/
├── manifest.yml
└── sources/
    ├── definitions/
    │   ├── pipeline.sql
    │   ├── access.sql
    │   └── dashboard.sql         ← DEFINE STREAMLIT statement
    └── streamlit/
        └── my_dashboard/         ← path referenced in FROM clause
            ├── streamlit_app.py
            ├── page_2.py
            ├── pyproject.toml
            └── snowflake.yml

Then add the DEFINE STREAMLIT statement to your DCM definitions with:

• The fully qualified name for the Streamlit object
• The relative path from the manifest to the Streamlit folder (always starting with sources/)
• The entrypoint file name (MAIN_FILE)
• The warehouse to use for query execution (QUERY_WAREHOUSE)
• The compute pool and runtime (COMPUTE_POOL, RUNTIME_NAME) for container-runtime apps
• An optional display title (TITLE)
• Any external access integrations (EXTERNAL_ACCESS_INTEGRATIONS) and stage imports (IMPORTS) the app requires

define streamlit DEMO{{env_suffix}}.SERVE.MY_DASHBOARD
    from 'sources/streamlit/my_dashboard'    -- relative path from manifest to Streamlit folder
    main_file = 'streamlit_app.py'
    query_warehouse = DEMO_WH{{env_suffix}}
    compute_pool = SYSTEM_COMPUTE_POOL_CPU
    runtime_name = SYSTEM$ST_CONTAINER_RUNTIME_PY3_11
    title = 'My Dashboard'
    external_access_integrations = ()
    imports = ()
;

Plan & deploy a DCM Project with a Streamlit app¶

Run your regular DCM plan and deploy commands. If either the DEFINE STREAMLIT statement in your definitions or the file hash for any of the files within the Streamlit app folder has changed since the last successful deployment, PLAN shows the Streamlit object as part of the changeset. DEPLOY replaces any modified files and creates a new version.

PLAN only validates that the Streamlit object can be created. It doesn’t test whether the app itself will run successfully when started.

After the first successful deployment, the Streamlit app is immediately live. DCM automatically initializes the live version after creating the Streamlit object, so you don’t need to run ALTER STREAMLIT manually.

Removing the DEFINE STREAMLIT statement drops the Streamlit object on the next deployment.

Functional limitations¶

• DCM Jinja templating variables are not passed through to Streamlit Python files. You can use Jinja in the DEFINE STREAMLIT statement itself (for example, to set the warehouse or compute pool name), but not inside your app code.

  • To reference environment-specific objects from inside your Streamlit app at runtime, query the active context using CURRENT_DATABASE(), CURRENT_SCHEMA(), or similar functions to infer the environment.

• Only relative paths to the Streamlit folder are supported. You can’t specify a path to another repo or folder outside of the DCM Project.

DEFINE DBT PROJECT¶

You can define a dbt project, its orchestration, infrastructure, and access control together in a single DCM Project folder, then deploy everything to any environment with one command.

Most commonly used is the combination of dbt projects + Tasks to execute dbt test and dbt run on a defined schedule. You can define a DAG of Tasks to orchestrate runs of different dbt projects or individual models.

Create a DCM Project for dbt¶

You can take your existing dbt project folder, which includes:

• models
• dbt_project.yml
• packages.yml
• profiles.yml

and move it inside the sources/ folder of your DCM Project, since the Snowflake CLI only uploads files from the sources/ folder hierarchy. Place it outside of sources/definitions/ and organize it in any subfolder structure you like, for example sources/dbt/dbt_pipeline/.

my_dcm_project/
├── manifest.yml
└── sources/
    ├── definitions/
    │   ├── pipeline.sql          ← DEFINE DBT PROJECT statement
    │   ├── access.sql
    │   └── infra.sql
    └── dbt/
        └── dbt_pipeline/         ← path referenced in FROM clause
            ├── dbt_project.yml
            ├── packages.yml
            ├── profiles.yml
            └── models/

Then add the DEFINE DBT PROJECT statement to your DCM definitions with:

• The relative path from the manifest to the dbt folder (always starting with sources/)
• A default target (which can use jinja templating to match the DCM deployment target)

In addition, you can add Tasks to execute dbt commands after the deployment as well as grants on the dbt project object or future tables and views.

define dbt project {{db}}.PROJECTS.DBT_PIPELINE
    from 'sources/dbt/dbt_pipeline'    --relative path from manifest to dbt folder
    default_target = '{{dbt_env}}'
;


define task {{db}}.PROJECTS.RUN_DBT_PIPELINE  -- optional: Task(s) to execute your deployed dbt project
    warehouse = {{wh}}
    schedule = '60 MINUTE'
    started     -- (optional) new DCM-specific property that defines the target-state after a DCM deployment
  as
    execute dbt project {{db}}.PROJECTS.DBT_PIPELINE args='run';

define task {{db}}.PROJECTS.TEST_DBT_PIPELINE
    warehouse = {{wh}}
    after {{db}}.PROJECTS.RUN_DBT_PIPELINE
    started
  as
    execute dbt project {{db}}.PROJECTS.DBT_PIPELINE args='test';

If you need to run dbt deps to get external packages, you can run CREATE NETWORK RULE IF NOT EXISTS and CREATE EXTERNAL ACCESS INTEGRATION IF NOT EXISTS in a DCM pre-hook (DCM Hooks are also part of this private preview).

Pass DCM variables to dbt¶

DCM Jinja templating and dbt templating variables are completely isolated. There’s no automatic pass-through between the DCM manifest.yml configuration and the dbt profiles.yml targets. The two configurations must be maintained separately and kept in sync.

If you need values from the DCM templating context inside a dbt run (for example, the active dbt target), pass them explicitly through the args of the EXECUTE DBT PROJECT statement. Jinja in args is rendered by DCM before the command is executed, so any DCM templating variable can be injected.

define dbt project DCM_DEMO_2_MARKETING{{env_suffix}}.PROJECTS.DBT_PIPELINE
    from 'sources/dbt/dbt_pipeline'
    default_target = '{{dbt_env}}'
;


-- Task graph to schedule dbt project test and run
define task DCM_DEMO_2_MARKETING{{env_suffix}}.PROJECTS.DBT_PIPELINE_RUN
    warehouse = DCM_DEMO_2_MARKETING_WH{{env_suffix}}
    schedule = '60 MINUTE'
    started
  as
    execute dbt project DCM_DEMO_2_MARKETING{{env_suffix}}.PROJECTS.DBT_PIPELINE
        args='run --target {{dbt_env}}'
;

Plan & deploy a DCM Project for dbt¶

Run your regular DCM plan and deploy commands. If the DEFINE DBT PROJECT statement in your definitions has changed since the last successful deployment, then PLAN will:

• Render the jinja templating
• Compile the entire DCM Project
• Show the dbt project as part of the plan output

The dbt project is compiled during DEPLOY, not during PLAN. PLAN only validates that the dbt project object can be created successfully. It doesn’t check whether the dbt project will run successfully.

Tables created by dbt don’t show as “DCM managed entities” because they aren’t defined directly in the DCM definitions. Removing the DEFINE DBT PROJECT statement drops the dbt project object on the next deployment, but it won’t drop the tables created by dbt.

You can also consider creating a new DCM Project for dbt on top of an existing “platform” project.

Functional limitations¶

• PLAN and DEPLOY output only show the operation for a Snowflake dbt project object (CREATE / ALTER / DROP) and don’t show more granular changes in the dbt project configuration or models.
• Dependencies: dbt models can refer to other objects defined in DCM, but other DCM objects can’t reference tables created by dbt, meaning dbt projects can’t have downstream dependencies.
• Currently, only relative paths to dbt project files are supported. You can’t specify a path to another repo or folder outside of the DCM Project.

CLI enhancements¶

An early-access version of the Snowflake CLI includes improvements to several DCM Projects commands. To install it:

uv tool install 'git+https://github.com/snowflakedb/snowflake-cli.git@dcm-early-access'

All of the commands below support the --save-output flag, which saves the command output as a .json file under out/.

snow dcm plan¶

• Shows a compressed file upload summary (file counter per path) with a progress bar
• Shows more granular changes for ALTER operations in the output

snow dcm deploy¶

• Shows a compressed file upload summary (file counter per path) with a progress bar
• Shows a progress bar for the PLAN and DEPLOY execution phases
• Shows more granular changes for ALTER operations in the output

snow dcm compile (new command)¶

snow dcm compile runs a static analysis of all DCM definitions and returns any errors or warnings found, grouped by file and entity. It’s intended for quickly checking iterative definition changes and catching errors before committing.

• Validates syntax and dependencies
• Runs faster than PLAN, but doesn’t catch all possible errors. (Always run PLAN to preview changes before deploying)
• Shows a compressed file upload summary (file counter per path) with a progress bar

snow dcm dependencies (new command)¶

snow dcm dependencies runs a static analysis of all DCM definitions and builds a Mermaid flowchart representing the dependencies between all objects in the project (tables, dynamic tables, views, functions, procedures, and tasks).

The diagram is written to out/dependencies.md. The CLI prints a link to the file so you can open it in your IDE’s Markdown preview and explore the dependency graph visually.

Note that these dependencies refer to the deployment of objects (CREATE). It does not resolve run-time dependencies (for example, a Task that calls a stored procedure).

• Generate a dependency diagram for the current project:
  Copy code

  snow dcm dependencies
  
  snow dcm dependencies --target dev
  

snow dcm refresh¶

• Shows updated output formatting

snow dcm test¶

• Shows updated output formatting

Revert to the official release¶

To revert to the latest official release on main:

uv tool uninstall snowflake-cli
uv tool install snowflake-cli

DDL Hooks¶

DDL Hooks are intended as an interim solution for defining integrations until they are supported natively in DCM using DEFINE statements. DDL Hooks do not offer full functional parity to regular DCM definitions.

Capabilities of DDL Hooks¶

• Each DCM Project can contain only 1 pre-hook
• The pre-hook can contain multiple DDL statements
• DDL statements inside the hook are executed in the order they are defined
• The hook supports only 2 types of commands:
• CREATE IF NOT EXISTS (recommended)
  • For one-time execution to create the object
  • Skipped any time an integration with this name already exists
• CREATE OR REPLACE
  • Executed at every DCM deployment
  • Use when the definition of an existing object has changed and should be replaced completely
• Only DDL statements are supported (no USE, no SET, no COPY INTO…)

Key advantages of DDL Hooks compared to custom SQL pre-scripts¶

1. DCM Hooks are plannable. Other DCM definitions can declare dependencies on objects created in the pre-hook.
   • Example: A Notification Integration defined in the pre-hook can be referenced by an Alert defined in DCM definitions.
2. DCM Hooks support Jinja templating, using the same variables as the rest of the DCM Project.

Functional limitations¶

• Create statements from DCM Hooks show as operations in the PLAN changeset and the deployment history, but don’t include granular details about their individual properties.
• Errors from executing DCM Hooks don’t show the exact error line and in some cases not the full stack-trace.
• Removing a DDL statement from a DCM Hook does NOT drop the object.
• Hooks can’t be defined inside Jinja loops, as it would create multiple pre-hooks. However, Jinja code including loops can be used inside a hook.

Comparison between DCM definitions, DDL Hooks, custom SQL pre-scripts¶

Supported object types for DDL Hooks¶

Only integration object types are supported:

• API Integration
• Notification Integration
• External Access Integration
• Catalog Integration
• Security Integration
• Storage Integration

Once these object types are supported natively with DCM DEFINE statements, the hook support will be deprecated.

Examples:

ATTACH PRE_HOOK
AS [

    CREATE API INTEGRATION IF NOT EXISTS GITHUB_API_{{env_suffix}}
        API_PROVIDER = git_https_api
        API_ALLOWED_PREFIXES = ('https://github.com')
        ALLOWED_AUTHENTICATION_SECRETS = all
        ENABLED = true;

    CREATE NOTIFICATION INTEGRATION IF NOT EXISTS DCM_EMAIL_NOTIFICATIONS_{{env_suffix}}
        TYPE = EMAIL
        ENABLED = true
        ALLOWED_RECIPIENTS = ('example@example.com');

];