Migrate from dbt Core to dbt Projects on Snowflake

This is a step-by-step guide for teams moving from standard dbt Core to dbt Projects on Snowflake. The goal is to take the dbt Core project you run today (in your local IDE, kicked off by an orchestrator such as Airflow) and get it running natively inside Snowflake, with as little change as possible. Most of your project files stay exactly the same.

This guide is written for admins who are comfortable copying and pasting SQL. Every step shows the SQL you can run, and where it helps, the equivalent clicks in Snowsight.

What changes (and what doesn’t)

Good news first: your models/, seeds/, macros/, tests/, dbt_project.yml, and packages.yml files don’t need to change. dbt is still dbt.

Here’s the short list of what’s different when dbt runs inside Snowflake:

Topicdbt Core todaydbt Projects on Snowflake
Where you edit and runLocal IDE + terminalSnowflake Workspaces (a web IDE in Snowsight) or Cortex Code Desktop
What kicks off runsThird-party orchestrator such as AirflowSnowflake tasks (scheduled SQL)
Connection / authprofiles.yml with account, user, passwordprofiles.yml in the project root, no account, user, or password needed
dbt engineWhatever you installedPick a Snowflake-managed runtime, for example 1.10.15 (dbt Core) or 2.0.0-preview (dbt Fusion). No installs.
Getting packages (dbt deps)Runs locallyRuns in Snowflake using an external access integration
Deployingn/aA dbt project object (a versioned snapshot of your code in Snowflake)

The six steps this guide walks through:

  1. Set up Workspaces (and pick the dbt runtime)
  2. (Optional) Set up PrivateLink to your Git server
  3. Connect a Git repository with OAuth2
  4. Move your profiles.yml into the project root
  5. Create an external access integration so dbt can read its packages
  6. Deploy the project and schedule it with a task

Related docs:

Step 1: Set up Workspaces

Workspaces are a web-based IDE inside Snowsight. They’re the easiest way to get a dbt Core project running on Snowflake: you can edit files, run dbt compile / dbt run / dbt build, see the DAG, and deploy, all from the browser. No local install required.

Related docs:

Pick the dbt runtime version

Snowflake runs your project on a managed dbt runtime. The version you choose decides the engine: a 1.x version runs dbt Core (Python), a 2.x version runs dbt Fusion (Rust). Pin to whichever version your team already has experience with.

You can set the default for the whole account so nobody has to specify it every time. This also sets the initial runtime version that Workspaces use:

-- Account-level default (requires an admin role). Workspaces use this as their initial runtime.

-- To default to dbt Core 1.10.15:
ALTER ACCOUNT SET DEFAULT_DBT_VERSION = '1.10.15';

-- Or, to default to dbt Fusion:
ALTER ACCOUNT SET DEFAULT_DBT_VERSION = '2.0.0-preview';

To see what versions are available at any time:

SELECT SYSTEM$SUPPORTED_DBT_VERSIONS();

You can always override the version on an individual project later (shown in step 6).

If you ever need to move other dbt Core projects to Fusion, see Migrate to dbt Fusion.

Related docs: Supported dbt Core versions

Things to know before you start

A few things that trip people up with Workspaces:

  • Workspaces are personal by default. Each user’s workspace lives in their own personal database and isn’t shared. If you want several people to collaborate in the same workspace, create a shared workspace in a regular database and schema instead. See Workspaces overview.
  • 20,000 file limit per dbt project object. This counts everything in the project directory, including the target/, dbt_packages/, and logs/ folders that dbt generates. Large projects with big package trees can bump into this. Contact your account representative if your team has a project larger than this threshold.
  • profiles.yml is required in each project folder (covered in step 4).
  • Public repos are read-only. If you connect a public Git repo, you can pull but you can’t commit and push back from the workspace. For your real project, connect it as a private repo with OAuth2 (step 3).
  • Fusion runs dbt deps automatically. When you’re on Fusion and your packages.yml lists packages but there’s no dbt_packages folder yet, Fusion quietly runs dbt deps during dbt compile / dbt run, which needs internet access. Workspaces make this painless: create the external access integration once (step 5), select it the first time, and it stays pre-selected for every command that might need it, so you don’t have to think about it again.

Note

Skip this entire step and go to step 3 unless your Git server is reachable only over PrivateLink (no public internet access). Most teams connect to their Git provider over the public internet and don’t need this step.

PrivateLink is a dedicated private network connection between Snowflake and your Git server, so Git traffic never crosses the public internet. Setting it up is a one-time admin task that must be done before you connect your repository in step 3, and it only works when Snowflake and your Git server are in the same cloud and region. Note that OAuth2 doesn’t work over PrivateLink, so on this path you authenticate with a token instead.

  1. On your cloud provider, create a private link service that accepts requests from Snowflake (see the walkthrough linked below).

  2. In Snowflake, provision the outbound private endpoint with your private link service ID and your Git server’s domain (AWS example shown; Azure and Google Cloud use their own service ID format):

    SELECT SYSTEM$PROVISION_PRIVATELINK_ENDPOINT(
  'com.amazonaws.vpce.us-west-2.vpce-svc-xxxxxxxx',   -- your private link service ID
  'git.example.com'                                   -- your Git server domain
);

  3. Accept the endpoint on your cloud provider, then check its status:

    SELECT SYSTEM$GET_PRIVATELINK_ENDPOINTS_INFO();

  4. Create the API integration with USE_PRIVATELINK_ENDPOINT = TRUE and token-based authentication (a secret holding a personal access token). Add TLS_TRUSTED_CERTIFICATES if your server uses a self-signed certificate:

    CREATE OR REPLACE SECRET git_pat_secret
  TYPE = password
  USERNAME = 'your-git-username'
  PASSWORD = 'your-personal-access-token';

CREATE OR REPLACE API INTEGRATION git_api_integration
  API_PROVIDER = git_https_api
  API_ALLOWED_PREFIXES = ('https://git.example.com/my-workspace')
  ALLOWED_AUTHENTICATION_SECRETS = (git_pat_secret)
  USE_PRIVATELINK_ENDPOINT = TRUE
  ENABLED = TRUE;

    For most providers, USERNAME is your actual Git username. For Bitbucket, set USERNAME literally to the string x-token-auth (a Bitbucket convention) and put your token in PASSWORD:

    -- Bitbucket example
CREATE OR REPLACE SECRET git_pat_secret
  TYPE = password
  USERNAME = 'x-token-auth'              -- literal value, not your username
  PASSWORD = 'your-bitbucket-access-token';

    After you finish this step, skip the OAuth options in step 3 and go straight to Create the Git-connected workspace. When you create the workspace, select Personal access token as the authentication method and point it at this secret.

Related docs:

Step 3: Connect your Git repository with OAuth2

Workspaces sync to a branch in your Git repo, so you keep your normal Git workflow (branches, commits, pull requests). The cleanest sign-in experience for interactive development is OAuth2: your team signs in to the Git provider once and Snowflake handles the rest, no tokens to paste or rotate.

Note

If you don’t intend to use Workspaces in your initial onboarding to dbt Projects on Snowflake, you can skip this section. Your team can still use the Snowflake CLI (shown in step 6) to deploy dbt project objects and schedule them.

The setup is two parts: an admin creates an API integration that tells Snowflake how to talk to your Git provider, and each user signs in with OAuth when they create a Git-connected workspace.

Tip

If you completed step 2 (PrivateLink), you already created your token-based API integration. Skip the OAuth options below and go straight to Create the Git-connected workspace.

Redirect URI (required for all providers except GitHub)

Most OAuth2 providers ask for a redirect URI (also called a callback URL) when you register an OAuth application. This is a fixed Snowflake URL based on your account’s region — it is not generated by the API integration. The pattern is:

https://apps-api.c1.<region>.<cloud>.app.snowflake.com/oauth/complete-secret

For example, an account in AWS US West (Oregon) uses:

https://apps-api.c1.us-west-2.aws.app.snowflake.com/oauth/complete-secret

You set this URL as the callback when you create the OAuth app, and the provider then gives you the client ID and client secret that you plug into the API integration. The GitHub option below skips this — the Snowflake GitHub App handles the redirect URI for you.

To let Snowflake build the whole URL for you, run this and copy the result:

SHOW REGIONS;
SELECT DISTINCT
  'https://apps-api.c1.' || "region" || '.' || "cloud" || '.app.snowflake.com/oauth/complete-secret'
    AS redirect_uri
FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
WHERE "snowflake_region" = SPLIT_PART(CURRENT_REGION(), '.', -1);

Create the API integration for your Git provider

Pick your Git provider to see the setup steps:

GitHub is the simplest case. Snowflake publishes a pre-built OAuth app (the Snowflake GitHub App), so you don’t register your own OAuth application or redirect URI. The admin just creates an API integration that points at it:

-- Run as an admin role (needs CREATE API INTEGRATION).
CREATE OR REPLACE API INTEGRATION git_api_integration
  API_PROVIDER = git_https_api
  API_ALLOWED_PREFIXES = ('https://github.com/my-org')   -- your GitHub org or account URL
  API_USER_AUTHENTICATION = (TYPE = SNOWFLAKE_GITHUB_APP)
  ENABLED = TRUE;

On first sign-in, GitHub asks an org admin to authorize the snowflakedb app; after that, everyone in the account can use it.

The Snowflake GitHub App is at github.com/apps/snowflakedb.

Create the Git-connected workspace

Once the API integration exists, anyone with USAGE on it can create a workspace from the repo:

  1. Sign in to Snowsight.
  2. In the navigation menu, select Projects > Workspaces.
  3. In the Workspaces menu, select From Git repository.
  4. Paste the repository URL (for example, https://github.com/my-org/analytics or https://gitlab.com/my-group/analytics).
  5. Give the workspace a name.
  6. Under API Integration, select git_api_integration.
  7. For the authentication method, select OAuth2, then Sign in and authorize Snowflake to access the repository. Grant read access to metadata and read/write access to code so you can pull and push.
  8. Select Create.

Snowflake clones the repo and opens the workspace with your dbt project files ready to edit.

Related docs:

Step 4: Move your profiles.yml into the project root

In dbt Core, your profiles.yml usually lives outside the project (in ~/.dbt/) and holds your account, user, and password. On Snowflake, you bring that file into the root of the dbt project folder, and you get to drop the sensitive bits.

Why it’s simpler: the project already runs inside Snowflake, under the signed-in user and account. So dbt doesn’t need an account, a user, or a password. The account and user keys can be left as empty or placeholder strings (dbt still expects the keys to exist), and there’s no password field at all.

Here’s a profiles.yml you can drop in. It defines a dev and a prod target:

my_project:
  target: dev
  outputs:
    dev:
      type: snowflake
      account: 'not needed'   # ignored on Snowflake; runs under the current account
      user: 'not needed'      # ignored on Snowflake; runs as the current user
      role: transformer
      database: dev_db
      schema: analytics
      warehouse: dbt_wh
      threads: 8
    prod:
      type: snowflake
      account: 'not needed'
      user: 'not needed'
      role: transformer
      database: prod_db
      schema: analytics
      warehouse: dbt_wh
      threads: 8

There’s no password, no authenticator, no key-pair path. That’s the point: nothing secret lives in the file.

Once profiles.yml is in the project root with at least one valid target, each target shows up in the Profile picker in the workspace toolbar. Pick a profile, pick a command (compile, run, build), and run it.

Related docs: Workspaces for dbt Projects on Snowflake

Step 5: Create an external access integration for dbt packages

If your project uses packages in packages.yml (for example, dbt-labs/dbt_utils), dbt needs to reach out to the internet to download them when it runs dbt deps. Snowflake blocks outbound network access by default, so you give dbt a narrow, allowlisted path with an external access integration.

This matters extra for Fusion: as noted in the gotchas, Fusion auto-runs dbt deps during compile/run when packages are declared but not yet downloaded. Without external access, that step fails with a network error. Setting this up once avoids that.

Run the following as an admin role:

-- 1. Allowlist the hosts dbt needs to download packages.
CREATE OR REPLACE NETWORK RULE dbt_network_rule
  MODE = EGRESS
  TYPE = HOST_PORT
  VALUE_LIST = (
    'hub.getdbt.com',       -- dbt Package hub
    'codeload.github.com'   -- packages hosted on GitHub
  );

-- 2. Wrap the rule in an external access integration that dbt can use.
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION dbt_ext_access
  ALLOWED_NETWORK_RULES = (dbt_network_rule)
  ENABLED = TRUE;

To populate packages in the workspace, select Deps from the command list, choose your external access integration, and run it. This creates the dbt_packages folder and a package-lock.yml.

If your team pulls packages from other hosts (a private Git package server, for example), add those hostnames to the VALUE_LIST.

Related docs:

Step 6: Deploy the project and schedule it with a task

Once the project runs cleanly in the workspace, you “ship” it by creating a dbt project object. Think of this as a versioned, read-only snapshot of your code that lives in a Snowflake database and schema. You then schedule it with a task (this replaces the Airflow job).

Deploy as a dbt project object

Using SQL. CREATE DBT PROJECT copies your code in and creates version 1. Point FROM at the workspace version of the project. Attach the external access integration from step 5 so Fusion can run dbt deps during compile, and pin the Fusion runtime:

CREATE OR REPLACE DBT PROJECT prod_db.analytics.my_dbt_project
  FROM 'snow://workspace/USER$.PUBLIC."my_dbt_workspace"/versions/live/my_dbt_project'
  DEFAULT_TARGET = 'prod'
  DBT_VERSION = '2.0.0-preview'
  EXTERNAL_ACCESS_INTEGRATIONS = (dbt_ext_access)
  COMMENT = 'Analytics dbt project';

Note

The FROM URL points at your workspace’s live version. The example uses a personal workspace (the default), whose path is USER$.PUBLIC."<workspace_name>". If you use a shared workspace, replace USER$.PUBLIC with the database and schema that contain the object (for example, prod_db.analytics."<workspace_name>"). The trailing segment (my_dbt_project) is the dbt project folder inside the workspace. The easiest way to get this exactly right is to deploy from the UI once and copy the CREATE DBT PROJECT SQL it prints.

Confirm it exists:

SHOW DBT PROJECTS IN DATABASE prod_db;

To push later code changes, add a new version instead of replacing the object:

ALTER DBT PROJECT prod_db.analytics.my_dbt_project
  ADD VERSION
  FROM 'snow://workspace/USER$.PUBLIC."my_dbt_workspace"/versions/live/my_dbt_project';

Using the UI. In the workspace, select Connect > Deploy dbt project, choose the target database and schema, choose Create dbt project, give it a name, optionally set a default target and the external access integration, then select Deploy. The Output tab shows the exact CREATE DBT PROJECT SQL it ran, so the UI and SQL paths produce the same thing.

Graduate to CI/CD. Deploying by hand is the right way to get going. Longer term, we recommend managing updates to the dbt project object through a CI/CD pipeline using the Snowflake CLI’s snow dbt commands, driven by a GitHub Action or GitLab pipeline. The shape is short and clean: on a pull request, deploy a copy to dev and test it; on merge, deploy the new version to production.

# CI — on each pull request: deploy a copy to dev, then run + test it
snow dbt deploy my_dbt_project --source ./my_dbt_project --database dev_db --schema analytics --external-access-integration dbt_ext_access
snow dbt execute my_dbt_project --database dev_db --schema analytics build --target dev

# CD — on merge to main: deploy the new version to the production project object
snow dbt deploy my_dbt_project --source ./my_dbt_project --database prod_db --schema analytics --external-access-integration dbt_ext_access

For the full setup (auth, secrets, workflow files), see CI/CD integrations on dbt Projects on Snowflake and the CI/CD tutorial.

Related docs:

Schedule it with a task

A task runs the EXECUTE DBT PROJECT command on a schedule. This is the piece that replaces Airflow. Two rules to remember:

  • The task must be in the same database and schema as the dbt project object.
  • Tasks that run dbt must use a user-managed warehouse (serverless tasks aren’t supported).

A daily run against the prod target:

CREATE OR ALTER TASK prod_db.analytics.run_dbt_project
  WAREHOUSE = dbt_wh
  SCHEDULE = 'USING CRON 0 6 * * * UTC'   -- every day at 06:00 UTC
AS
  EXECUTE DBT PROJECT prod_db.analytics.my_dbt_project
    ARGS = 'run --target prod';

You can chain a test run right after the run finishes:

CREATE OR ALTER TASK prod_db.analytics.test_dbt_project
  WAREHOUSE = dbt_wh
  AFTER prod_db.analytics.run_dbt_project
AS
  EXECUTE DBT PROJECT prod_db.analytics.my_dbt_project
    ARGS = 'test --target prod';

Tasks are created in a suspended state. Resume them to start the schedule (resume the child first, then the root):

ALTER TASK prod_db.analytics.test_dbt_project RESUME;
ALTER TASK prod_db.analytics.run_dbt_project RESUME;

You can also run a single execution by hand at any time:

EXECUTE DBT PROJECT prod_db.analytics.my_dbt_project ARGS = 'run --select my_model --target prod';

Using the UI. From the dbt project menu, select Create schedule, set the frequency, operation (run), profile, and any flags (for example, --select customer_metrics). Snowflake creates the same CREATE TASK for you.

Related docs:

Where to go next