ML Lineage: Trace ML data flow¶

Limitations¶

• Tables and views created from model predictions do not currently capture the lineage relationship back to the model.

• Lineage information is not replicated at this time.

Snowflake intends to address these limitations in future releases of ML Lineage.

Required privileges¶

Users need the VIEW LINEAGE privilege to explore lineage from Python APIs. This privilege is automatically granted to the ACCOUNTADMIN role, which can then grant it to other roles at the account level. For example:

USE ROLE ACCOUNTADMIN;
GRANT VIEW LINEAGE ON ACCOUNT TO ROLE test_role;

Creating ML Lineage¶

Generally, Snowflake records lineage information when objects are created. Lineage for models is captured when the model is logged to the Model Registry. Training a model using Snowpark ML automatically generates lineage records if the model is trained from a Snowpark DataFrame.

Other scenarios, such as those listed below, can also generate lineage records with a little extra effort.

• Training a model using Snowpark MLfrom some other kind of data source (such as a pandas DataFrame).

• Training a model without using Snowpark ML or a Snowpark DataFrame.

• Training a model outside of Snowflake.

In these scenarios, you can still associate the source data object and the trained model by passing a Snowpark DataFrame backed by the source data object as sample_data to the Model Registry’s log_model method, as shown below.

registry.log_model(...,
          sample_input_data=df_backed_by_source_table)

Querying ML Lineage¶

You can query the lineage of ML artifacts in several ways.

Snowsight UI¶

Every artifact’s landing page has a Lineage tab. The default view displays upstream and downstream objects one step away from the selected object. For a more detailed exploration of lineage within the Snowsight UI, see Data Lineage in Snowsight.

A sample of the Snowsight view of lineage data is shown below.

Snowpark ML lineage API¶

The lineage method available on FeatureView, ModelVersion, and Dataset objects retrieves lineage relationships for the current object, so you can trace the lineage of data objects retrieved from the Snowflake Feature Store or Model Registry.

For all supported objects, the lineage method accepts the following arguments:

• direction, either UPSTREAM or DOWNSTREAM. DOWNSTREAM is the default.

• domain_filter, a list of target object types for which lineage will be retrieved. The default is to return all lineage relationships. The available domains are "feature_view", "dataset", "model", "table", and "view".

The method returns a list of connected lineage nodes. These nodes can be instances of Dataset, FeatureView, or ModelVersion, if you have imported these classes into your Python session. Otherwise, each node is represented by a generic LineageNode instance.