Snowpark Connect Next Generation File Reader¶

The next generation file reader speeds up file ingestion for Snowpark Connect for Spark by running Spark’s own read semantics on warehouse-native I/O. Because Spark format options, schema inference, and type handling are inherited rather than reimplemented, it reduces or eliminates the need to rewrite your Spark applications. No code changes are required — it works once your account is enabled for the preview.

This page is a step-by-step guide to enable and test the reader. You can run the test two ways: in a Snowflake Notebook (Path A) or through the Spark Job API (Code Bundle) (Path B).

Schema inference sampling¶

The next generation reader follows OSS Spark behavior for schema inference sampling. In OSS Spark, schema inference is controlled by the samplingRatio option — the fraction of rows examined, which defaults to 1.0 (all rows); when set to less than 1.0, it randomly samples rows across all partitions of the target file. The next generation reader therefore scans the full file by default, which reduces the chance of inference errors but increases inference time.

The current Snowpark Connect file reader takes a different approach. It exposes a Snowpark Connect-specific option, spark.read.option("rowsToInferSchema", <n>), that caps how many rows per file it samples when inferring a schema. The default is 20000: it types each column from the first 20,000 rows and stops. This speeds up inference but introduces a risk of errors — for example, a column that looks like INT for the first 20,000 rows but holds a non-numeric value afterward. Unlike the next generation reader, which samples random rows when samplingRatio is less than 1.0, the current reader always samples the first n rows set by rowsToInferSchema. Increasing the rowsToInferSchema sample size reduces inference speed.

In summary, the next generation file reader samples the full file by default for schema inference, whereas the current file reader samples only 20,000 rows by default. To benchmark them on inference performance, align the sampling size on both. For an equal-grounds comparison, disable the current reader’s per-file cap:

# Current reader: disable the 20,000-row sampling cap for an equal-grounds comparison
df = (spark.read
      .option("inferSchema", "true")
      .option("rowsToInferSchema", 0)   # 0 = no per-file cap; infer from all rows, like the next generation reader / OSS Spark
      .csv("@my_stage/data/"))

If you trust the dataset to be uniformly typed across all fields, or for exploration and prototyping, you can set samplingRatio to less than 1.0 in the next generation reader to speed up schema inference.

Parquet: before you start¶

The direct Parquet reader has additional restrictions in private preview:

• External stages only. The direct Parquet reader is enabled only for named external stages (@stage over S3, Azure, or GCS backed by a storage integration). Reads from internal, user, or table stages, key-based cloud URLs, file://, or mixed sources silently fall back to the current reader. To set up an external stage, see Create an S3 stage (and CREATE STAGE for Azure or GCS).
• Don’t use it on Parquet written by Snowflake yet. Snowflake-written Parquet encodes integer columns as DECIMAL, and the direct reader’s strict type validation rejects that mismatch with server error 100496 (for example, Parquet Type 'DECIMAL', Expected Type 'INT'). Point it at externally-produced Parquet for now.

Path A: Snowflake Notebook (Container Runtime)¶

Create a notebook in a workspace¶

In Snowsight, open Workspaces and create a new notebook. Notebooks in workspaces run on Snowflake Container Runtime. When you first run it, you create a notebook service. Configure the following:

• Python version (3.10–3.12) and Container Runtime version.
• A CPU compute pool (your role needs USAGE on it, and the pool must allow the NOTEBOOK workload).
• Under Artifact Repository, select snowflake.snowpark.pypi_shared_repository (Snowflake’s managed PyPI repo).

For notebook setup details, see Notebooks in workspaces.

Install snowpark-connect 1.41.0 and verify¶

Private preview requires snowpark-connect 1.41.0 or later. In the first cell, install it:

!pip install --no-deps snowpark-connect

Restart the kernel (Kernel » Restart), then verify in a new cell. The printed version must be 1.41.0 or later:

import importlib.metadata
print(importlib.metadata.version("snowpark-connect"))   # must be 1.41.0 or later

Start a Spark session and enable the reader¶

from snowflake.snowpark_connect import init_spark_session
spark = init_spark_session()

# CSV and JSON: next generation file reader
spark.conf.set("snowflake.file.nextGenReader.enabled", "true")

# Parquet: direct Parquet reader (separate flag)
spark.conf.set("snowpark.connect.parquet_direct.enabled", "true")

Read a file to test¶

Point Spark at your files in a stage and run your existing read code. The new reader works with your current CSV and JSON workloads with no rewrites. You can toggle the flag back to "false" to compare behavior and performance against the current reader.

Confirm the read used the new reader¶

To verify the read actually used the next generation file reader (and not the old COPY INTO path), check recent account-level query history for the new reader’s table functions:

SELECT query_id, start_time, LEFT(query_text, 200) AS query_text
FROM TABLE(<your_db>.INFORMATION_SCHEMA.QUERY_HISTORY(
       END_TIME_RANGE_START => DATEADD('minute', -15, CURRENT_TIMESTAMP()),
       RESULT_LIMIT => 5000))
WHERE query_text ILIKE '%INFER_STAGE_FILE_SCHEMA%'         -- CSV/JSON next-gen reader
   OR query_text ILIKE '%STAGE_FILE_READER%'               -- CSV/JSON next-gen reader
   OR query_text ILIKE '%PARQUET_DIRECT_EXTERNAL_STAGE%'   -- Parquet direct reader
ORDER BY start_time DESC;

Seeing INFER_STAGE_FILE_SCHEMA or STAGE_FILE_READER in the generated SQL confirms the read went through the next generation reader. If instead you see COPY INTO or INFER_SCHEMA, the old reader was used — double-check that the configuration flag from the previous step was set on the active session.

For Parquet, the direct reader emits a CREATE … TEMPORARY ICEBERG TABLE … PARQUET_DIRECT_EXTERNAL_STAGE='<db.schema.stage>' … statement; seeing PARQUET_DIRECT_EXTERNAL_STAGE confirms it was used. If you instead see the VARIANT/INFER_SCHEMA path (SELECT $1:"col"::type … FROM '@stage'(FILE_FORMAT=>…parquet…)), the read fell back to the current reader — check the flag and that the source is an external stage.

Path B: Spark Job API (Code Bundle)¶

You can also run the test through the Spark Job API (Code Bundle).

Set up and submit a Spark job (Code Bundle)¶

For how to set up and submit a Spark job on Snowflake through the Code Bundle API, follow Submit Spark jobs on Snowflake (via Code Bundles).

Enable the reader in your Spark application¶

As in Path A, set the Spark configuration in your Spark application before reading.

from snowflake.snowpark_connect import init_spark_session
spark = init_spark_session()

# CSV and JSON: next generation file reader
spark.conf.set("snowflake.file.nextGenReader.enabled", "true")

# Parquet: direct Parquet reader (separate flag)
spark.conf.set("snowpark.connect.parquet_direct.enabled", "true")

Confirm the read used the new reader¶

Use the same account-level query history check as in Path A: Confirm the read used the new reader.