External lineage¶

External lineage extends Snowflake’s native lineage to include external data sources and destinations, providing you with visibility into data flows across your entire data ecosystem. It captures lineage from external ETL tools and source databases to create a unified view of how data moves through your data pipeline.

Lineage can be captured between a Snowflake object and an external object, and also between two external objects. Because an event isn’t required to include a Snowflake object, a pipeline that moves data through several external systems is represented as a connected chain even where no Snowflake object sits between them. External lineage captures lineage between individual columns as well as between objects.

OpenLineage is an open standard for capturing and sharing data lineage information across diverse data tools and platforms. Snowflake leverages this framework by accepting OpenLineage-compatible events through a REST endpoint. External tools like dbt and Apache Airflow can use the endpoint to send lineage metadata to Snowflake, which then incorporates this information into the native lineage graph displayed in Snowsight.

External lineage REST endpoint

Copy code

/api/v2/lineage/external-lineage

Snowflake base URL for REST endpoints

Copy code

https://<account_identifier>.snowflakecomputing.com

Where account-identifier is the account identifier of your Snowflake account. You can use either the account name format or the account locator format as your account identifier.

For example, if your account identifier is myorg-dev_account, then the base URL of the external lineage endpoint is: https://myorg-dev_account.snowflakecomputing.com

External lineage workflow¶

Implementing external lineage for a data tool consists of the following tasks:

1. Grant the necessary privileges to the user who is authenticating to the external lineage endpoint, and make sure that user’s role can access the Snowflake objects your lineage events reference.
2. Configure your data tool to send OpenLineage events to the Snowflake REST endpoint.
3. Choose an authentication method that works for Snowflake REST APIs, and then configure your data tool to use it to authenticate its requests to the external lineage endpoint.
4. Use your data tool as usual. OpenLineage events are sent to Snowflake automatically and appear in the native lineage graph in Snowsight.

If you want to test the external lineage endpoint before you configure a data tool to emit OpenLineage events, see Send manual requests to establish lineage.

View your data lineage¶

To view data lineage in Snowsight, complete the following steps:

1. Sign in to Snowsight with the necessary privileges.
2. In the navigation menu, select Catalog » Database Explorer, and then select a supported object such as a table or view.
3. Select the Lineage tab.

When a data tool sends lineage information to Snowflake, external objects appear in the Snowsight lineage graph and are labeled as an external node. For example:

You can select an external object or the line connecting objects to obtain additional information just like you can with native lineage.

Grant Snowflake privileges¶

After a REST request is authenticated, Snowflake checks whether the user associated with the request is authorized to use external lineage. The user associated with the request must have a role that is granted the INGEST LINEAGE privilege on the account.

For example, suppose you want requests sent by the service user dbt_integration_user to show up in Snowsight lineage. As an administrator, run the following commands to create a dedicated role, grant it the necessary privilege, and then grant the role to the user:

CREATE ROLE dbt_lineage_role;
GRANT INGEST LINEAGE ON ACCOUNT TO ROLE dbt_lineage_role;
GRANT ROLE dbt_lineage_role TO USER dbt_integration_user;

Access to referenced Snowflake objects¶

The INGEST LINEAGE privilege authorizes a role to use external lineage, but it doesn’t grant access to the objects named in a lineage event. The role sending the request must also be able to resolve every Snowflake object referenced in the inputs and outputs properties of the payload. Snowflake resolves these objects when it receives the request, so the role needs the privileges required to see each object: typically USAGE on the database and schema that contain it, along with a privilege such as SELECT on the object itself.

Snowflake resolves the objects using the role that is in effect for the request, which is the default role of the user associated with the request. Granting the required privileges to a role that the user is granted, but that isn’t in effect for the request, isn’t sufficient. If the user has no default role, or its default role isn’t the role you granted the privileges to, set the default role explicitly.

If any Snowflake object in the payload can’t be resolved, either because the object doesn’t exist or because the role can’t see it, Snowflake rejects the whole event with a 400 HTTP status code and error code 394919, and stores no lineage from that event. Because a single unresolvable object rejects the entire event, the other objects in the same event don’t produce lineage either.

Building on the previous example, the following commands make dbt_lineage_role the default role of the user and let it resolve the objects in the my_db.my_schema schema:

ALTER USER dbt_integration_user SET DEFAULT_ROLE = dbt_lineage_role;
GRANT USAGE ON DATABASE my_db TO ROLE dbt_lineage_role;
GRANT USAGE ON SCHEMA my_db.my_schema TO ROLE dbt_lineage_role;
GRANT SELECT ON ALL TABLES IN SCHEMA my_db.my_schema TO ROLE dbt_lineage_role;
GRANT SELECT ON FUTURE TABLES IN SCHEMA my_db.my_schema TO ROLE dbt_lineage_role;

Configure your data tool¶

The following sections provide basic instructions for using external lineage with dbt and Apache Airflow.

• Configure dbt to send lineage data to Snowflake
• Configure Airflow to send lineage data to Snowflake

Configure dbt to send lineage data to Snowflake¶

The following steps provide the minimum configuration you need to set up your dbt environment. Consult the OpenLineage dbt documentation and the OpenLineage specification to configure your OpenLineage-dbt integration.

1. Install the OpenLineage-dbt integration:

   Copy code

   pip3 install openlineage-dbt
   

2. Set your transport variables to specify the base URL, endpoint, and security token for external lineage.

   For example, if the account identifier of your account is MYORG-DEV_ACCOUNT, define the following code in your YAML configuration file:

   Copy code

   transport:
   type: http
   url: https://MYORG-DEV_ACCOUNT.snowflakecomputing.com
   endpoint: /api/v2/lineage/external-lineage
   auth:
      type: api_key
      apiKey: eyJ0eXAiOiJKV1QiLsecuritytoken...
   compression: gzip
   

3. Replace dbt commands with dbt-ol. For example, change the dbt run command to dbt-ol run.

   These dbt-ol commands are required by the OpenLineage-dbt integration, and aren’t unique to Snowflake.

For more information about OpenLineage-dbt integrations, including other methods of setting variables, see the OpenLineage dbt documentation.

Configure Airflow to send lineage data to Snowflake¶

The following steps provide the minimum configuration you need to set up your Airflow environment for Airflow version 2.7+, which is the preferred version for OpenLineage. Consult the OpenLineage Airflow documentation and the OpenLineage specification to configure your OpenLineage-Airflow integration.

1. Install the OpenLineage Airflow integration for version 2.7+:

   Copy code

   pip install apache-airflow-providers-openlineage
   

   If you use an older version of Airflow, install openlineage-airflow instead.

2. Set your transport variables to specify the base URL, endpoint, and security token for external lineage.

   For example, if the account identifier of your account is MYORG-DEV_ACCOUNT, define the following code in your YAML configuration file:

   Copy code

   transport:
   type: http
   url: https://MYORG-DEV_ACCOUNT.snowflakecomputing.com
   endpoint: /api/v2/lineage/external-lineage
   auth:
      type: api_key
      apiKey: eyJ0eXAiOiJKV1QiLsecuritytoken...
   compression: gzip
   

For more information about OpenLineage-Airflow integrations, including other methods of setting variables, see the OpenLineage Airflow documentation.

Choose an authentication method¶

Snowflake provides multiple ways to authenticate requests to a Snowflake REST endpoint like the one used by external lineage. For a complete list of authentication methods, see Authenticating Snowflake REST APIs with Snowflake.

After you select your preferred authentication method, you must generate a security token for a specific user. The token is used to associate a user with the REST request so that Snowflake can authenticate the user and verify that the user is authorized to use external lineage.

After successfully associating a user with a security token in Snowflake, you need to configure your data tool to authenticate its requests with this token. For example, if you use a YAML configuration file to set OpenLineage transport variables, use the following code to specify the security token that is sent in the header of the request:

transport:
   auth:
      type: api_key
      apiKey: eyJ0eXAiOiJKV1QiLsecuritytoken...

For other methods of specifying a security token, see the OpenLineage documentation for your data tool.

Send manual requests to establish lineage¶

External lineage works by accepting JSON payloads that conform to the OpenLineage specification for COMPLETE events. When integrated with a data tool, the tool emits these COMPLETE events. But you can also construct a COMPLETE event, then send it to the endpoint by using any tool or language that can send POST requests to an endpoint.

A valid request consists of the following method, base URL, and endpoint:

POST https://<account_identifier>.snowflakecomputing.com/api/v2/lineage/external-lineage

Where account_identifier is the account identifier of your Snowflake account.

The following example shows how to use curl to send lineage information to external lineage:

curl -i -X POST \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLsecuritytoken..." \
 -H "Accept: application/json" \
 -H "User-Agent: myApplicationName/1.0" \
 -H "X-Snowflake-Authorization-Token-Type: KEYPAIR_JWT" \
 -d "@request_body.json" \
 "https://MYORG-DEV_ACCOUNT.snowflakecomputing.com/api/v2/lineage/external-lineage"

Where request_body.json conforms to the OpenLineage specification for COMPLETE events. For more information about this JSON payload, see Payload requirements.

Authentication and authorization of a manual request¶

The authentication and authorization of a manual request sent to the external lineage endpoint are the same as those in a request sent from a data tool.

• The header of the request must include a security token from one of the forms of authentication supported by Snowflake REST endpoints.
• The user associated with the security token must have the proper privileges.

Payload requirements¶

When you send the JSON payload in a manual request to the external lineage endpoint, the payload must meet the following requirements:

• Must conform to the OpenLineage specification.

• Must be a COMPLETE event. That is, the eventType property must be COMPLETE. Events with any other eventType are rejected with a 400 HTTP status code.

• The inputs property and outputs property can reference Snowflake objects, external objects, or both. Lineage between two external objects is supported; an event is no longer required to include at least one Snowflake object.

• Every Snowflake object referenced in the inputs and outputs properties must exist and must be visible to the role sending the request. See Access to referenced Snowflake objects.

• Must contain the following properties:

  • inputs
  • outputs
  • eventType
  • eventTime
  • job

  You can optionally include the run property, which is useful in identifying the job. The payload can contain additional properties, but Snowflake ignores them.

Minimal payload example¶

The following example shows a minimal payload that you can send to the external lineage endpoint:

{
   "eventType": "COMPLETE",
   "eventTime": "2025-03-12T06:51:12.000Z",
   "job": {"namespace": "exampleNamespace", "name": "exampleJob"},
   "run": {"runId": "123e4567-e89b-12d3-a456-426614174000"},
   "producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
   "schemaURL": "https://openlineage.io/spec/0-0-1/OpenLineage.json",
   "inputs": [{"namespace": "snowflake://AXORG-AX_TEST_PP8", "name": "OL_TEST.OL_TEST_SCH.TEST_DEMO"}],
   "outputs": [{"namespace": "postgres://localhost:5432", "name": "PDB.SCH.OUTPUT"}]
}

Lineage between two external objects¶

Neither the input nor the output has to be a Snowflake object. The following example establishes lineage between two external systems, so a pipeline that moves data between them is represented even though no Snowflake object is involved:

{
   "eventType": "COMPLETE",
   "eventTime": "2025-03-12T06:51:12.000Z",
   "job": {"namespace": "exampleNamespace", "name": "exampleJob"},
   "run": {"runId": "123e4567-e89b-12d3-a456-426614174000"},
   "producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
   "schemaURL": "https://openlineage.io/spec/0-0-1/OpenLineage.json",
   "inputs": [{"namespace": "s3://example-bucket", "name": "raw/orders"}],
   "outputs": [{"namespace": "postgres://db.company.com:5432", "name": "public.orders_staging"}]
}

Specifying object types¶

Within the outputs array of the payload, you can use the facets field to specify the type of the object, which can be any user-defined string. For example, the following snippet of the payload specifies that the object is of type VIEW:

"outputs": [
    {
        "namespace": "postgres://db.company.com:5432",
        "name": "db.schema.view",
        "facets": {"datasetType": {"datasetType": "VIEW"}},
    },
],

If you don’t specify a facets field, the type of object defaults to External Node.

Specifying multiple inputs¶

If a payload includes more than one input, the resulting lineage shows the output as a downstream object of both inputs. For example, if a payload has input A and B along with an output C, then the lineage shows both A-C and B-C.

Specifying column lineage¶

External lineage captures lineage between individual columns as well as between objects. To establish column lineage, use the columnLineage facet of an output dataset. The facet maps each output column to the input columns that it is derived from:

"outputs": [
    {
        "namespace": "postgres://db.company.com:5432",
        "name": "db.schema.orders_summary",
        "facets": {
            "columnLineage": {
                "fields": {
                    "TOTAL_AMOUNT": {
                        "inputFields": [
                            {
                                "namespace": "snowflake://MYORG-DEV_ACCOUNT",
                                "name": "SALES_DB.PUBLIC.ORDERS",
                                "field": "AMOUNT"
                            }
                        ]
                    }
                }
            }
        }
    }
]

Each key in the fields object is the name of a column in the output dataset, and each entry in its inputFields array identifies a source column by the namespace and name of its dataset plus the field name of the column.

Keep the following in mind when you specify column lineage:

• The namespace and name of each inputFields entry must match a dataset that is also listed in the inputs property of the same event. An entry that refers to a dataset outside the event’s inputs is ignored.
• Column mappings are captured on a best-effort basis. If a column can’t be resolved, Snowflake skips that mapping and continues. A column might not resolve because it was dropped or renamed, or because the role sending the request can’t see it. The event isn’t rejected, and the object-level lineage is still recorded.

Send requests to remove lineage¶

You can send a DELETE request to the external lineage endpoint to remove lineage that was established between a Snowflake object and an external object.

• To break lineage between the source object and target object, use URL query parameters to specify details about the two objects.
• To break lineage between an object and all of its downstream objects, specify the source object without specifying a target object.
• To remove a target object from the lineage graph regardless of how many objects are upstream of it, specify the target object without specifying a source object.

A valid request to remove lineage consists of the following method, base URL, and endpoint:

DELETE https://<account_identifier>.snowflakecomputing.com/api/v2/lineage/external-lineage

Access control for removing lineage¶

The user sending a request to remove lineage between objects must have the DELETE LINEAGE privilege on the account.

Limitations and considerations¶

• Snowflake doesn’t support OpenLineage version 2.
• The retention period for external lineage events is one year.
• Snowflake only accepts COMPLETE lineage events. Events with a different eventType, for example START or FAIL, are rejected with a 400 HTTP status code. Data tools that emit a full event lifecycle therefore receive 400 responses for their non-COMPLETE events. This is expected and doesn’t indicate a problem with the COMPLETE events that the same tool sends.
• Every Snowflake object referenced in an event must exist and be visible to the role sending the request, or the event is rejected. See Access to referenced Snowflake objects.
• Lineage from external sources doesn’t appear in the output of the GET_LINEAGE function. The GET_LINEAGE function returns native Snowflake lineage only.
• The fully qualified name of a dataset — that is, the input or output — can’t exceed 1000 characters.
• A single event can’t produce more than 15,000 lineage edges. An event produces one edge for each combination of input and output, so an event with 100 inputs and 200 outputs produces 20,000 edges and is rejected.
• You can’t store more than 20,000 external lineage edges in the same account. If you reach this limit, you must delete edges before adding new ones.