dbt Projects on Snowflake

dbt Core is an open-source data transformation tool and framework that you can use to define, test, and deploy SQL transformations.

With dbt Projects on Snowflake, you can use familiar Snowflake features to create, edit, test, run, and manage your dbt Core projects. Deploying a dbt project object gives you built-in observability, CI/CD integration, and an up-to-date dbt project object details page in Snowsight. The typical workflow is as follows:

  1. Start with a valid dbt project: Your project needs a dbt_project.yml, profiles.yml, and model files. Store it in a workspace in Snowsight or a Git repository connected to Snowflake. If you’re migrating an existing dbt Core project, see Migrate from dbt Core to dbt Projects on Snowflake.

  2. Install dependencies: Execute the dbt deps command within a Snowflake workspace, local machine, or git orchestrator to populate the dbt_packages folder for your dbt Project.

    For more information, see Understand dependencies for dbt Projects on Snowflake.

  3. Deploy the DBT PROJECT object: Create a schema-level DBT PROJECT object by copying your project files into a new version of that object. You can do this by using the CREATE OR REPLACE DBT PROJECT … FROM <source> command or the snow dbt deploy Snowflake CLI command.

    For more information, see Deploy dbt project objects.

  4. Execute the deployed dbt project object: Execute a dbt project within a dbt project object by using the EXECUTE DBT PROJECT command or the snow dbt execute Snowflake CLI command. Executing a dbt project involves invoking dbt commands to build or test models; this is what you schedule and orchestrate.

    For more information, see EXECUTE DBT PROJECT and snow dbt execute commands.

  5. Schedule and orchestrate: Use Snowflake tasks to schedule dbt project object executions natively, or integrate with external orchestrators like Apache Airflow.

    For more information, see Schedule execution of dbt project objects on Snowflake and Understanding orchestration for dbt Projects on Snowflake.

  6. Set up CI/CD integrations: Use the Snowflake CLI to integrate deployment and execution into your CI/CD workflows.

    For a complete walkthrough, see Tutorial: Set up CI/CD integrations on dbt Projects on Snowflake. For the CI/CD reference, see CI/CD integrations on dbt Projects on Snowflake.

  7. Monitor the dbt project object: Use Snowflake monitoring features to inspect, manage, and tune dbt project execution whether you execute a dbt project object manually or use tasks to execute dbt project objects on a schedule.

    For more information, see Monitor dbt Projects on Snowflake.

For best practices for data engineering teams planning to adopt dbt Projects on Snowflake, see Best practices for dbt Projects on Snowflake.

Key concepts

  • dbt project objects: A dbt project is a directory that contains a dbt_project.yml file and a set of files that define dbt assets, such as models and sources. A DBT PROJECT is a schema-level object that contains versioned source files for your dbt project in Snowflake. You can connect a dbt project object to a workspace, or you can create and manage the object independent of a workspace. You can CREATE, ALTER, and DROP dbt project objects like other schema-level objects in Snowflake.

    A dbt project object is typically based on a dbt project directory that contains a dbt_project.yml file. This is the pattern that Snowflake uses when you deploy (create) a dbt project object from within a workspace.

    A workspace is a development environment, not a dbt project object. When you deploy from a workspace, Snowflake copies your workspace project files into a new or existing dbt project object located in your target database and schema.

    For more information, see Understand dbt project objects. For the benefits of creating a dbt project object, see Benefits of creating a dbt project object.

  • Schema customization: dbt uses the default macro generate_schema_name to decide where a model is built. You can customize how dbt builds your models, seeds, snapshots, and test tables.

    For more information, see Understand schema generation and customization.

  • Workspaces: Workspaces in the Snowflake web interface are a Git-connected web IDE where you can visualize, test, run, and scaffold one or many dbt projects, link them to a Snowflake dbt project object to create/update it, and edit other Snowflake code in one place.

    For more information, see Workspaces for dbt Projects on Snowflake.

  • Versioning: Every dbt project object is versioned; versions live under snow://dbt/<db>.<schema>.<project>/versions/....

    For more information, see Versions for dbt project objects and files.

  • Supported dbt versions: Snowflake provides managed runtimes for specific dbt Core and dbt Fusion versions. You can pin a version at creation time, override it per execution, or set an account-level default.

    For more information, see Supported dbt versions for dbt Projects on Snowflake.

  • Supported commands: dbt project objects support a subset of dbt commands and flags.

    For more information, see Supported dbt commands and flags.

  • Costs: Executing a dbt project object uses a virtual warehouse and incurs standard compute costs. There are no additional licensing or per-user fees.

    For more information, see Understanding costs for dbt Projects on Snowflake.