개인 식별 정보(PII) 수정

PII includes names, addresses, phone numbers, email addresses, tax identification numbers, and other data that can be used (alone or with other information) to identify an individual. Most organizations have regulatory and compliance requirements around handling PII data. AI_REDACT is a fully-managed Cortex AI Function that helps you help redact PII from unstructured text data, using a large language model (LLM) hosted by Snowflake to identify PII and replace it with placeholder values.

AI_REDACT는 콜 센터 코칭, 감정 분석, 보험 및 의료 분석, ML 모델 학습 등 기타 사용 사례를 위한 텍스트를 준비하는 데 도움이 될 수 있습니다.

팁

AI_REDACT를 적용하기 전에 AI_PARSE_DOCUMENT 또는 AI_TRANSCRIBE를 사용하여 문서나 음성 데이터를 텍스트로 변환합니다.

AI_REDACT

AI_REDACT 함수는 입력 텍스트의 개인 식별 정보(PII)를 자리 표시자 값으로 대체합니다.

중요

AI_REDACT는 AI 모델을 사용하여 최선의 방식으로 수정 작업을 수행합니다. 항상 출력을 검토하여 조직의 데이터 개인정보 보호정책을 준수하는지 확인하세요. AI_REDACT가 데이터의 PII를 수정하지 못한 경우 Snowflake에 알려주시기 바랍니다.

리전 가용성

리전 가용성 섹션을 참조하십시오.

제한 사항

  • 수정은 AI 모델을 사용하여 수행되며, 모든 개인 식별 정보를 찾지 못할 수도 있습니다. 항상 출력을 검토하여 조직의 데이터 개인정보 보호정책을 준수하는지 확인하세요. AI_REDACT가 특정 PII를 수정하지 못한 경우 Snowflake에 알려주시기 바랍니다.

  • COUNT_TOKENS 및 AI_COUNT_TOKENS 함수는 아직 AI_REDACT를 지원하지 않습니다.

  • 현재 AI_REDACT는 올바른 형식의 영어 텍스트에서 가장 잘 작동합니다. 다른 언어나 맞춤법, 구두점 또는 문법 오류가 많은 텍스트에서는 성능이 다를 수 있습니다.

  • AI_REDACT currently redacts only US PII and some UK and Canadian PII, where noted in PII 카테고리 감지.

  • AI_REDACT is currently limited in the number of tokens it can input and output. Input and output together can be up to 4,096 tokens. Output is limited to 1,024 tokens. If the input text is longer, split it into smaller chunks and redact each chunk separately, perhaps using SPLIT_TEXT_RECURSIVE_CHARACTER. See Chunking example for an example of redacting text that exceeds token limits.

    참고

    토큰은 AI 모델에서 처리하는 가장 작은 데이터 단위입니다. 영어 텍스트의 경우 업계 가이드라인에서는 토큰 1개를 약 4자 또는 0.75단어로 간주합니다.

PII 카테고리 감지

AI_REDACT supports redacting the following categorise of PII. The values in the Category column are the strings supported in the optional categories argument.

카테고리

참고

NAME

Recognizes full name, first name, middle name, and last name

EMAIL

PHONE_NUMBER

DATE_OF_BIRTH

GENDER

Recognizes male, female, and nonbinary

AGE

ADDRESS

Identifies:

  • complete postal address (US, UK, CA)

  • street address (US, UK, CA)

  • postal code (US, UK, CA)

  • city (US, UK, CA)

  • state (US) or province (CA)

  • county, borough, or township (US)

NATIONAL_ID

Identifies Social Security numbers (US)

PASSPORT

Identifies passport numbers (US, UK, CA)

TAX_IDENTIFIER

개인 납세자 번호(ITNs) 식별

PAYMENT_CARD_DATA

Identifies complete card information, card number, expiration date, and CVV

DRIVERS_LICENSE

Supported US, UK, CA

IP_ADDRESS

참고

AI_REDACT supports partial matches for some PII categories. For example, a first name alone is sufficient to trigger redaction with the [NAME] placeholder.

Error handling

Ordinarily, AI_REDACT raises an error if it cannot process the input text. When a query redacts multiple rows, an error causes the entire query to fail. To allow processing to continue with other rows, you can set the session parameter AI_SQL_ERROR_HANDLING_USE_FAIL_ON_ERROR to FALSE. Errors then return NULL instead of stopping the query.

ALTER SESSION SET AI_SQL_ERROR_HANDLING_USE_FAIL_ON_ERROR=FALSE;
Copy

With this parameter set to FALSE, you can also pass TRUE as the final argument to AI_REDACT, which causes the return value to be an OBJECT that contains separate fields for the redacted text and any error message. One of these fields is NULL depending on whether the AI_REDACT call processed successfully.

비용 고려 사항

AI_REDACT incurs costs based on the number of input and output tokens processed, as with other Cortex AI Functions. See the Snowflake Pricing Guide for details.

예

기본 예제

다음 예에서는 입력 텍스트에서 이름과 주소를 수정합니다.

SELECT AI_REDACT(
    input => 'My name is John Smith and I live at twenty third street, San Francisco.'
);
Copy

출력:

My name is [NAME] and I live at [ADDRESS]

The following example redacts only names and email addresses from the input text. Note that the text only contains a first name, which is recognized and redacted as [NAME]. The input text does not contain an email address, so no email placeholder appears in the output.

SELECT AI_REDACT(
    input => 'My name is John and I live at twenty third street, San Francisco.',
    categories => ['NAME', 'EMAIL']
);
Copy

출력:

My name is [NAME] and I live at twenty third street, San Francisco.

엔드투엔드 예제

다음 예에서는 한 테이블의 행을 처리하고 수정된 출력을 다른 테이블에 삽입합니다. 유사한 접근 방식을 사용하여 기존 테이블의 열에 수정된 데이터를 저장할 수 있습니다.

수정 후 텍스트는 AI_SENTIMENT에 전달되어 전체 감정 정보를 추출합니다.

-- Create a table with unredacted text
CREATE OR REPLACE TABLE raw_table AS
  SELECT 'My previous manager, Washington, used to live in Kirkland. His first name was Mike.' AS my_column
  UNION ALL
  SELECT 'My name is William and I live in San Francisco. You can reach me at (415).450.0973';

-- view unredacted data
SELECT * FROM raw_table;

-- Create a redaction table
CREATE OR REPLACE TABLE redaction_table (
  value VARCHAR
);

-- Redact PII from raw_table and insert into redaction_table
INSERT INTO redaction_table
SELECT AI_REDACT(my_column) AS value FROM raw_table;

-- view redacted results
SELECT * FROM redaction_table;

-- Run AI_SENTIMENT on redacted text
SELECT
    value AS redacted_text,
    AI_SENTIMENT(value) AS summary_sentiment
FROM redaction_table;
Copy

오류 처리 예제

This example, based on the preceding example, shows how to handle errors when processing multiple rows with AI_REDACT. It sets the session parameter AI_SQL_ERROR_HANDLING_USE_FAIL_ON_ERROR and passes TRUE as the last argument to AI_REDACT. This causes the function to return an OBJECT with separate fields for the redacted text and any error message, one of which is NULL depending on whether the function succeeded or failed.

ALTER SESSION SET AI_SQL_ERROR_HANDLING_USE_FAIL_ON_ERROR=FALSE;

-- Create a redaction table with columns for value and error message
CREATE OR REPLACE TABLE redaction_table (
  value VARCHAR,
  error VARCHAR
);

-- Redact PII from raw_table and insert into redaction_table
-- Both the redacted text and any error message are stored
INSERT INTO redaction_table
SELECT
  result:value::STRING AS value,
  result:error::STRING AS error
  FROM (SELECT AI_REDACT(my_column, TRUE) AS result FROM raw_table);
Copy

Chunking example

This example illustrates how to redact PII from long text by splitting the text into smaller chunks, redacting each chunk separately, and then recombining the redacted chunks into the final output. This approach works around AI_REDACT’s token limits.

CREATE OR REPLACE TABLE patients (
    patient_id INT PRIMARY KEY,
    patient_notes text
);

CREATE OR REPLACE TABLE final_temp_table AS
WITH chunked_data AS (
    -- Step 1: Split text into chunks
    SELECT
        patient_id,
        chunk.value AS chunk_text,
        chunk.index AS chunk_index
    FROM
        patients,
        LATERAL FLATTEN(
            input => SNOWFLAKE.CORTEX.SPLIT_TEXT_RECURSIVE_CHARACTER(
                patient_notes,
                'none',
                1000
            )
        ) AS chunk
    WHERE
        patient_notes IS NOT NULL
        AND LENGTH(patient_notes) > 0
),
redacted_chunks AS (
    -- Step 2: Apply AI_REDACT to each chunk
  SELECT
  patient_id,
        chunk_index,
        chunk_text,
        TO_VARIANT(results:value) AS redacted_chunk,
        TO_VARIANT(results:error) AS error_string
        from (
    SELECT
        patient_id,
        chunk_index,
        chunk_text,
        AI_REDACT(chunk_text,TRUE) AS results
    FROM
        chunked_data
)
),

-- Step 3: Concatenate redacted chunks
final AS (
SELECT
chunk_text as original,
IFF(error_string IS NOT NULL, chunk_text, redacted_chunk) AS redacted_text,
patient_id,
chunk_index
FROM
    redacted_chunks
)
SELECT * FROM final;

SELECT
  patient_id,
  LISTAGG(redacted_text, '') WITHIN GROUP (ORDER BY chunk_index) AS full_output
  FROM final_temp_table
  GROUP BY patient_id;
Copy
언어: 한국어