Validation and error handling¶

This topic describes how the Snowflake Connector for Kafka validates data and handles errors during ingestion.

Validation modes¶

Start with server-side validation (snowflake.validation=server_side, the default). Use it for standard Snowflake tables, including pipelines that don’t need schema evolution, and for Snowflake-managed Iceberg tables.

Choose a mode based on the target table and your type or error-routing requirements. Using Avro or Protobuf alone doesn’t require client-side validation:

• Standard Snowflake table, no connector-side type or error-routing requirement: Use server_side.
• Standard Snowflake table, new columns should use declared schema types: Use client_side, subject to the type mappings described in Client-side validation, or pre-create the columns and use server_side.
• Standard Snowflake table, client-detected validation errors must go to a Kafka dead-letter queue: Use client_side.
• Snowflake-managed Iceberg table: Start with server_side. A pre-created Iceberg v3 table with a compatible schema can also use client-side evolution; see Client-side schema evolution for Iceberg. Iceberg auto-creation with snowflake.autocreate.table.type=iceberg requires server_side.

The snowflake.validation configuration property controls the mode. Schema evolution also requires the appropriate table settings and privileges; selecting a validation mode alone doesn’t enable it.

Server-side validation (default)¶

snowflake.validation=server_side

With server-side validation, the connector doesn’t validate rows against the target table’s column types before sending them to Snowflake. Kafka converters still deserialize records and can enforce the source schema. The Snowflake backend performs validation and schema evolution, aligned with the behavior of COPY and Snowpipe.

• Invalid records are captured in the Snowflake Error Table associated with the target table.
• The DLQ isn’t used for ingestion validation errors in this mode. Use Error Tables instead. However, records that fail to be deserialized by the Kafka converters (before reaching Snowflake) are still routed to the DLQ if configured.
• Delivers maximum throughput by offloading validation from the connector.
• Supports both default pipe and user-defined pipe modes.

When to use server-side validation:

• You want the default mode for a standard Snowflake table, including when schema evolution isn’t needed.
• You’re ingesting into a Snowflake-managed Iceberg table.
• You want maximum throughput and Snowflake error tables for rejected records.
• You’re using user-defined pipes for in-flight transformations.

Client-side validation¶

snowflake.validation=client_side

With client-side validation, the connector validates data types and schema compatibility before sending rows to Snowflake.

• Invalid records are handled according to the errors.tolerance setting (see below).
• With schema evolution enabled, the connector runs DDL to add new top-level columns or drop NOT NULL constraints when a field is missing or null. It doesn’t widen existing column types or evolve nested structured fields.
• Supports Dead Letter Queue (DLQ) for routing invalid records.
• Only works with the default pipe ({tableName}-STREAMING). If a user-created pipe exists and client-side validation is enabled, the connector fails with ERROR_5030.

When the converter supplies a populated Kafka Connect STRUCT schema, client-side evolution maps its field types to Snowflake column types. Without that schema, it infers types from incoming values. Registry registration alone doesn’t alter the table; evolution is triggered by fields in incoming records.

Declared types aren’t preserved exactly. For example, FLOAT32 maps to FLOAT, FLOAT64 to DOUBLE, bytes to BINARY, and the Kafka Connect logical decimal type to VARCHAR. Decimal precision and scale aren’t used to create a NUMBER column. Structs and maps map to VARIANT; arrays map to unstructured ARRAY. Pre-create columns when you need exact types or precision.

Client-side evolution requires ENABLE_SCHEMA_EVOLUTION=TRUE and a direct EVOLVE SCHEMA or OWNERSHIP grant to the connector role. The client validator doesn’t support structured OBJECT, ARRAY, or MAP column schemas. If validation initialization fails, the connector logs a warning and disables client-side validation for that channel; connector-side evolution doesn’t run.

When to use client-side validation:

For a table with column types supported by the client validator:

• You need new columns to use the connector’s mappings of declared schema types rather than value-based server inference.
• You need client-detected validation errors routed to a Kafka dead-letter queue. Errors detected only by Snowflake aren’t routed there.
• You deliberately retain connector-side validation when migrating a v3 pipeline.

Error handling¶

Error tolerance¶

The errors.tolerance property controls how the connector responds to errors:

errors.tolerance=none (default)

The connector task fails on the first error. If a DLQ is configured, the failing record is still sent to the DLQ before the task aborts.

errors.tolerance=all

The connector continues ingesting data. Invalid records are routed to the DLQ (if configured) or silently dropped.

Warning

Setting errors.tolerance=all without configuring a DLQ topic causes invalid records to be silently dropped. This is an unsafe configuration that can cause data loss.

Dead Letter Queue (DLQ)¶

To configure a DLQ:

errors.deadletterqueue.topic.name=my_topic_dlq
errors.log.enable=true

The following records are routed to the DLQ (if configured):

• Converter errors (both validation modes): Records that fail to be deserialized by the Kafka converters are sent to the DLQ regardless of the validation mode. This applies with both errors.tolerance=none (record pushed to DLQ, then task fails) and errors.tolerance=all (record pushed to DLQ, task continues).
• Client-side validation errors: Records that fail client-side data conversion or schema validation are sent to the DLQ when snowflake.validation=client_side.

Records that fail Snowflake server-side ingestion aren’t routed to the DLQ. Use Error Tables instead.

Error tables (server-side validation)¶

When using snowflake.validation=server_side, records that fail server-side ingestion (type mismatches, constraint violations) are logged in the Error Table for inspection and replay.

For more information about Error Tables and error handling with Snowpipe Streaming, see Error handling in Snowpipe Streaming high-performance architecture.

Channel status inspection (server-side validation)¶

When using server-side validation, the connector inspects the Snowpipe Streaming channel status before committing Kafka offsets. If the channel reports an increasing rows_error_count:

• With errors.tolerance=none: The connector raises a fatal error (ERROR_5030) and the task fails.
• With errors.tolerance=all: The connector continues, logging the error count. Records after the invalid row can still be ingested. If Error Tables are enabled, the invalid records are stored there.

With client-side validation, the connector detects invalid records before sending them to Snowflake. Invalid records either cause an immediate task failure (errors.tolerance=none) or are routed to the DLQ (errors.tolerance=all).

To investigate broken records with server-side validation, review the channel history and use the gap-finding technique described in Detect and recover from errors using metadata offsets. Kafka offset information needed for this technique is available in the RECORD_METADATA column.