External lineage¶

External lineage workflow¶

Implementing external lineage for a data tool consists of the following tasks:

1. Grant the necessary privileges to the user who is authenticating to the external lineage endpoint.

2. Configure your data tool to send OpenLineage events to the Snowflake REST endpoint.

3. Choose an authentication method that works for Snowflake REST APIs, and then configure your data tool to use it to authenticate its requests to the external lineage endpoint.

4. Use your data tool as usual. OpenLineage events are sent to Snowflake automatically and appear in the native lineage graph in Snowsight.

If you want to test the external lineage endpoint before you configure a data tool to emit OpenLineage events, see Send manual requests to establish lineage.

View your data lineage¶

To view data lineage in Snowsight, complete the following steps:

1. Sign in to Snowsight with the necessary privileges.

2. In the navigation menu, select Catalog » Database Explorer, and then select a supported object such as a table or view.

3. Select the Lineage tab.

When a data tool sends lineage information to Snowflake, external objects appear in the Snowsight lineage graph and are labeled as an external node. For example:

You can select an external object or the line connecting objects to obtain additional information just like you can with native lineage.

Grant Snowflake privileges¶

After a REST request is authenticated, Snowflake checks whether the user associated with the request is authorized to use external lineage. The user associated with the request must have a role that is granted the INGEST LINEAGE privilege on the account.

For example, suppose you want requests sent by the service user dbt_integration_user to show up in Snowsight lineage. As an administrator, run the following commands to create a dedicated role, grant it the necessary privilege, and then grant the role to the user:

CREATE ROLE dbt_lineage_role;
GRANT INGEST LINEAGE ON ACCOUNT TO ROLE dbt_lineage_role;
GRANT ROLE dbt_lineage_role TO USER dbt_integration_user;

Configure your data tool¶

The following sections provide basic instructions for using external lineage with dbt and Apache AirFlow.

• Configure dbt to send lineage data to Snowflake

• Configure Airflow to send lineage data to Snowflake

Choose an authentication method¶

Snowflake provides multiple ways to authenticate requests to a Snowflake REST endpoint like the one used by external lineage. For a complete list of authentication methods, see Authenticating Snowflake REST APIs with Snowflake.

After you select your preferred authentication method, you must generate a security token for a specific user. The token is used to associate a user with the REST request so that Snowflake can authenticate the user and verify that the user is authorized to use external lineage.

After successfully associating a user with a security token in Snowflake, you need to configure your data tool to authenticate its requests with this token. For example, if you use a YAML configuration file to set OpenLineage transport variables, use the following code to specify the security token that is sent in the header of the request:

transport:
   auth:
      type: api_key
      apiKey: eyJ0eXAiOiJKV1QiLsecuritytoken...

For other methods of specifying a security token, see the OpenLineage documentation for your data tool.

Send manual requests to establish lineage¶

External lineage works by accepting JSON payloads that conform to the OpenLineage specification for COMPLETE events. When integrated with a data tool, the tool emits these COMPLETE events. But you can also construct a COMPLETE event, then send it to the endpoint by using any tool or language that can send POST requests to an endpoint.

A valid request consists of the following method, base URL, and endpoint:

POST https://<account_identifier>.snowflakecomputing.com/api/v2/lineage/external-lineage

Where account_identifier is the account identifier of your Snowflake account.

The following example shows how to use curl to send lineage information to external lineage:

curl -i -X POST \
 -H "Content-Type: application/json" \
 -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLsecuritytoken..." \
 -H "Accept: application/json" \
 -H "User-Agent: myApplicationName/1.0" \
 -H "X-Snowflake-Authorization-Token-Type: KEYPAIR_JWT" \
 -d "@request_body.json" \
 "https://MYORG-DEV_ACCOUNT.snowflakecomputing.com/api/v2/lineage/external-lineage"

Where request_body.json conforms to the OpenLineage specification for COMPLETE events. For more information about this JSON payload, see Payload requirements.

{
   "eventType": "COMPLETE",
   "eventTime": "2025-03-12T06:51:12.000Z",
   "job": {"namespace": "exampleNamespace", "name": "exampleJob"},
   "run": {"runId": "123e4567-e89b-12d3-a456-426614174000"},
   "producer": "https://github.com/OpenLineage/OpenLineage/blob/v1-0-0/client",
   "schemaURL": "https://openlineage.io/spec/0-0-1/OpenLineage.json",
   "inputs": [{"namespace": "snowflake://AXORG-AX_TEST_PP8", "name": "OL_TEST.OL_TEST_SCH.TEST_DEMO"}],
   "outputs": [{"namespace": "postgres://localhost:5432", "name": "PDB.SCH.OUTPUT"}]
}

"outputs": [
    {
        "namespace": "postgres://db.company.com:5432",
        "name": "db.schema.view",
        "facets": {"datasetType": {"datasetType": "VIEW"}},
    },
],

Send requests to remove lineage¶

You can send a DELETE request to the external lineage endpoint to remove lineage that was established between a Snowflake object and an external object.

• To break lineage between the source object and target object, use URL query parameters to specify details about the two objects.

• To break lineage between an object and all of its downstream objects, specify the source object without specifying a target object.

• To remove a target object from the lineage graph regardless of how many objects are upstream of it, specify the target object without specifying a source object.

A valid request to remove lineage consists of the following method, base URL, and endpoint:

DELETE https://<account_identifier>.snowflakecomputing.com/api/v2/lineage/external-lineage

Limitations and considerations¶

• A Snowflake object must be either the INPUT or the OUTPUT of a COMPLETE event. That is, external lineage doesn’t ingest lineage events when neither the input data nor the output data is a Snowflake object.

• Snowflake doesn’t support OpenLineage version 1.

• The retention period for external lineage events is one year.

• Snowflake only recognizes COMPLETE lineage events. All other events emitted by a data tool are ignored.

• Lineage from external sources doesn’t appear in the output of the GET_LINEAGE function.

• External lineage doesn’t support the lineage of columns.

• The fully qualified name of a dataset — that is, the input or output — can’t exceed 1000 characters.

• You can’t store more than 10,000 events in the same account. If you reach this limit, you’ll have to delete events before adding new ones.