Classify sensitive data automatically¶

Get started¶

The basic workflow to automatically classify sensitive data consists of the following:

1. Create a classification profile that controls how often sensitive data in a schema is automatically classified, including whether system tags should be automatically applied after classification.

2. Optionally, use the classification profile to map user-defined tags to system tags so a column with sensitive data can be associated with a user-defined tag based on its classification.

3. Optionally, add a custom classifier to the classification profile so sensitive data can be automatically classified with user-defined semantic and privacy categories.

4. Set the classification profile on a schema so that tables in the schema get automatically classified.

For end-to-end examples of this workflow, see Examples.

About classification profiles¶

A data engineer creates a classification profile by creating an instance of the CLASSIFICATION_PROFILE class to define the criteria that are used to automatically classify tables in a schema. This criteria includes:

• How long a table should exist before automatically classifying it.

• How long before previously classified tables should be reclassified.

• Whether system and custom tags are automatically set on columns after the classification. You can decide whether you want Snowflake to automatically apply suggested tags or prefer to review proposed tag assignments, then apply them yourself.

• A mapping between system classification tags and user-defined object tags so the user-defined tags can be applied automatically.

When the data engineer assigns the classification profile to a schema, sensitive data in the tables of the schema are automatically classified on the schedule defined by the profile. A data engineer can assign the same classification profile to multiple schemas, or can create multiple classification profiles if there is a need to set different classification criteria for different schemas.

The process of automatically classifying data requires access to the raw data in the table. The raw data includes tables that have a masking policy assigned to a column. However, Snowflake preserves the intention of regulating access to protected data by using an internal role to automatically classify data. The internal role can access data protected by a masking policy, but this role is not accessible to users.

For an example of using the CREATE CLASSIFICATION_PROFILE command to create a classification profile, see Examples.

About tag mapping¶

You can use the classification profile to map SEMANTIC_CATEGORY system tags to one or more user-defined tags. This tag mapping allows a column with sensitive data to be automatically assigned a user-defined tag based on its classification. The tag map can be added while creating the classification profile or later by calling the <classification_profile_name>!SET_TAG_MAP method.

Because user-defined object tags can have a masking policy associated with them, you can use a tag map to enable automatic tag-based masking. If you choose to automatically apply tags after classification, you can automate the entire process of protecting columns with a masking policy based on the classification of data. As new data is added to a schema, the tag-based masking policies will be automatically assigned to the columns that contain sensitive data.

Regardless of whether you are defining the tag map while creating the classification profile or after, the contents of the map are specified as a JSON object. This JSON object contains the 'column_tag_map' key, which is an array of objects that specify a user-defined tag, the string value of that tag, and the semantic categories to which the tag is being mapped. After the tag map is associated with a classification profile and you automatically classify tables in a schema, the tag is assigned to the columns that correspond to the semantic categories.

The following is an example of a tag map:

'tag_map': {
  'column_tag_map': [
    {
      'tag_name':'tag_db.sch.pii',
      'tag_value':'Highly Confidential',
      'semantic_categories':[
        'NAME',
        'NATIONAL_IDENTIFIER'
      ]
    },
    {
      'tag_name': 'tag_db.sch.pii',
      'tag_value':'Confidential',
      'semantic_categories': [
        'EMAIL'
      ]
    }
  ]
}

Based on this mapping, if you have a column of email addresses and the classification process determines that the column contains these addresses, the tag_db.sch.pii = 'Confidential' tag is set on the column containing the email addresses.

If your tag map includes multiple JSON objects that map tags, tag values, and category values, the order of the JSON objects determines which tag and value to set on the column if there is a conflict. Specify the JSON objects in the desired assignment order from left to right, or top to bottom if you are formatting JSON.

If there is a conflict with a manually assigned tag and a tag applied by automatic classification, an error occurs. For information about tracking these errors, see Troubleshooting.

Implementing automatic custom classification¶

Snowflake lets you define custom classifiers that use custom logic to identify and classify sensitive data. For example, you can create a custom classifier that uses a regular expression to identify ICD-10 codes and classify them as belonging to the semantic category ICD_10_CODES.

After you’ve created a custom classifier, you can add it to the classification profile so that Snowflake automatically classifies data based on its logic. You can add the custom classifier when creating the classification profile or by calling the <classification_profile_name>!SET_CUSTOM_CLASSIFIERS method.

Adding both custom classifiers and a tag map in your classification profile provides a powerful governance solution. It allows you to automatically classify data based on your knowledge of what is sensitive and apply a user-defined tag that you can track. If you use this user-defined tag to implement tag-based masking, your domain-specific sensitive data is automatically protected by a masking policy as data is added to a schema.

View results of automatic classification¶

You can view the results of automatic classification in the following ways:

• Call the SYSTEM$GET_CLASSIFICATION_RESULT stored procedure. For example:

  CALL SYSTEM$GET_CLASSIFICATION_RESULT('mydb.sch.t1');
  

  Copy

  You cannot return results until the classification process completes. The automatic classification process does not start until one hour after setting the classification profile on the schema.

• Use a role that is granted the SNOWFLAKE.GOVERNANCE_VIEWER database role to query the DATA_CLASSIFICATION_LATEST view. For example:

  SELECT * FROM snowflake.account_usage.data_classification_latest;
  

  Copy

  Results might not appear until three hours after classification completes.

Limitations¶

• Classification profiles cannot be set on a reader account.

• You cannot automatically classify views.

• Only one classification profile can be set on a schema.

• The same classification profile cannot be set on more than 10,000 schemas.

• A maximum of 100 million tables can be classified in a schema.

• You cannot automatically classify a table if it has any of the following characteristics:

  • Has more than 10,000 columns.

  • Has a column with a name that has more than 255 characters.

  • Has a column with a name that includes the $ character.

  • Is from a share.

Access control¶

This section describes the privileges and roles that let you work with classification profiles and enable automatic sensitive data classification.

For an example of granting these privileges and database roles to the role of a data engineer, see Basic example: Automatically classifying tables in a schema.

Cost of automatically classifying sensitive data¶

Automatic sensitive data classification consumes credits as it uses serverless compute resources to classify tables in the schema. For more information about pricing for this consumption, see Table 5 in the Snowflake Service Consumption Table.

You can query views in the ACCOUNT_USAGE and ORGANIZATION_USAGE schemas to determine how much was spent on automatically classifying sensitive data. To monitor credit consumption, query the following views:

METERING_HISTORY view (ACCOUNT_USAGE)

Lets you retrieve the hourly cost of automatic classification by focusing on SENSITIVE_DATA_CLASSIFICATION in the SERVICE_TYPE column. For example:

SELECT
  service_type,
  start_time,
  end_time,
  entity_id,
  name,
  credits_used_compute,
  credits_used_cloud_services,
  credits_used,
  budget_id
  FROM snowflake.account_usage.metering_history
  WHERE service_type = 'SENSITIVE_DATA_CLASSIFICATION';

Copy

METERING_DAILY_HISTORY view (ACCOUNT_USAGE and ORGANIZATION_USAGE)

Lets you retrieve the daily cost of automatic classification by focusing on SENSITIVE_DATA_CLASSIFICATION in the SERVICE_TYPE column. For example:

SELECT
  service_type,
  usage_date,
  credits_used_compute,
  credits_used_cloud_services,
  credits_used
  FROM snowflake.account_usage.metering_daily_history
  WHERE service_type = 'SENSITIVE_DATA_CLASSIFICATION';

Copy

USAGE_IN_CURRENCY_DAILY (ORGANIZATION_USAGE)

Lets you retrieve the daily cost of automatic classification by focusing on SENSITIVE_DATA_CLASSIFICATION in the SERVICE_TYPE column. Use this view to determine the cost in currency, not credits.

Examples¶

• Basic example: Automatically classifying tables in a schema

• Example: Using a tag map and custom classifiers

• Example: Testing a classification profile before enabling automatic classification

Troubleshooting¶

By default, Snowflake uses the user event table to log events related to the automatic classification of sensitive data. If you want prevent classification events from being logged, set the ENABLE_AUTOMATIC_SENSITIVE_DATA_CLASSIFICATION_LOG account parameter to FALSE.

You can use the following query to access error messages from the event table:

SELECT
  record_type,
  record:severity_text::string log_level,
  parse_json(value) error_message
  FROM log_db.log_schema.log_table
  WHERE record_type='LOG' and scope:name ='snow.automatic_sensitive_data_classification'
  ORDER BY log_level;

The following are possible error messages, where the output is truncated to contain the "failure_reason" key and its value (the error message):

"failure_reason":"NO_TAGGING_PRIVILEGE"

"failure_reason":"MANUALLY_APPLIED_VALUE_PRESENT"

"failure_reason":"TAG_NOT_ACCESSIBLE_OR_AUTHORIZED"

For more information about event table messages, see Viewing log messages.