Snowpark Connect for Spark release notes for 2025¶

New features¶

• Support any type as output or input type in the Scala map and flatmap functions.

• Support joinWith.

• Support any return type in Scala UDFs.

• Support registerJavaFunction.

Bug fixes¶

• Fix JSON schema inference issue for JSON reads from Scala.

• Change return types of functions returning incorrect integral types.

• Fix update fields bug with struct type.

• Fix unbounded input decoder.

• Fix struct function when the argument is unresolved_star.

• Fix column name for Scala UDFs when the proto contains no function name.

• Add support for PATTERN in Parquet format.

• Handle error and errorIfExists write modes.

Improvements¶

None.

New features¶

• Bump snowflake-connector-python to <4.2.0.

• Add basic support for single-column map and flatMap operations on Scala datasets.

• Iceberg writing support TargetFileSize and PartitionBy.

Bug fixes¶

• Make SAS server initialization synchronous.

• Use snowpark-connect-deps-1==3.56.3.

• Fix saveAsTable with input_filename columns.

• Remove duplicated reading of the cache in Scala UDFs.

• Increase recursion limit.

• Fix format_number.

• Fix infer schema when query is provided in JDBC read.

• Only lock dict operation in cache.py to improve performance.

• Fix grouped data tests.

• Throw more detailed errors on table and read/write operations.

Improvements¶

None.

New features¶

• Introduce reduce function for Scala.

Improvements¶

None.

Bug fixes¶

• Fix failing array insert for nullable elements.

• Throw correct error on non-numeric args in covariance.

New features¶

• Support filter on a simple (single column) Dataset.

• Support Azure scheme URL parsing and special character file name.

Bug fixes¶

• Fix “Dataframe has no attribute dataframe” error in Scala catalog API.

• Fix aliases in subquery, document not working subqueries.

• Fix plan_id resolution after joins.

• Fix meta.yaml for multi-py versions.

• Enable use_vectorized_scanner as map type from parquet file was error.

• CSV reading inferSchema option specify datatype.

• Fix substr function handling of negative length.

• Use cached file formats in read_parquet.

• Improve local relation performance.

• Generate summary _common_metadata for parquet files.

• Remove repetitive setSchema, setRole, etc, for Snowflake pushdown.

Improvements¶

None.

New features¶

• Relax version requirements for grpcio and aiobotocore.

Improvements¶

• Specify dependencies version in meta.yaml.

• Build compiled and architecture-specific conda package.

• Ensure all CloudPickleSerializer.loads are not done in TCM.

• Include OSS SQL tests that start with the WITH clause.

• Do not upload Spark jars when running the server for pyt.

• Update internal queries count.

Bug fixes¶

• Fix tests for tcm.

• Fix CSV column name discrepancy from Spark.

• Use type cache for empty frames.

• Resolve Windows OSS runner general issues.

Improvements¶

• Generate unique workload names.

Bug Fixes¶

• Fix staged file reading.

New features¶

• Add parameter for view creation strategies.

• Support string <-> year month interval.

• Support multiple pivot columns and aliases for pivot values in Spark SQL.

• Integrate OpenTelemetry span and traces.

Improvements¶

None.

Bug fixes¶

• Add a trailing slash for remove command.

• Invalid GROUP BY issue with aggregation function and nilary functions.

• Notebook exceeds gRPC maximum message size.

• Fix temporary view creation with colliding names.

• array_size with null argument.

• Fix for $.0 JSON array access in get_json_object function.

• Fix self ANTI and SEMI LEFT joins.

• Handle different types in SQL function range.

• Fixed temporary view describe.

New features¶

• Add rowToInferSchema for CSV reading.

• Support INSERT INTO with CTE SQL command.

• I/O changes to add _SUCCESS file generation and metadata file filtering.

• update(submit): Support installing Snowpark Connect for Spark in the Snowpark Submit client container.

Improvements¶

None.

Bug fixes¶

• Fix _SUCCESS path update.

• Throw error on remove failure update.

• Sequence function supporting integral types inputs.

• Fix types in empty CreateTempViewUsing.

• Fix Parquet file repartitioning on write.

• Resolve aliases in ORDER BY clause correctly.

• Remove scope temp session parameter.

• Fixed multiple self joins with join condition.

• Fix column name resolution in pivot.

• SQL parser aware of session timezone.

• Interval type coercion with other types.

• Fix having with nested CTEs.

• Improve qualified name resolution in Spark.

New features¶

• Add script to run on the output from Git action for merging SQLs.

• Add --rebuild-whl parameter to notebook test runner.

• Add support for both qualifiers after join.

Improvements¶

None.

Bug fixes¶

• Support escape parameter in SQL LIKE commands.

• Overwrite bug in partitions.

• Validate column count on INSERT.

• Incompatibility for pow with NAN.

• Cross JOIN with condition.

• Column attribution logic in nested queries.

• Update error message for interval test.

• String type coercion in set operation UNION and EXCEPT, coerce NUMERIC, DATE, DATETIME to STRING.

• Correctly resolve Snowpark columns after a full outer self JOIN.

• Expression in aggregate function might be zero improvement.

• Update: Revert “[SCOS GA BUG] string type coercion in set opera”

• DataFrame union of decimal type columns now widen as necessary.

• String type coercion in set operation UNION and EXCEPT, coerce NUMERIC, DATE, DATETIME to STRING (part1).

• Object not existed issue in TCM.

• Fix to_binary(x, 'hex') where x has odd number of letters and digits.

• Fix joins with empty tables.

• Fix HAVING clause to prioritize grouping columns over aggregate aliases with same name.

New features¶

• Support for RepairTable.

• Make jdk4py an optional dependency of Snowpark Connect for Spark to simplify configuring Java home for end users.

• Support more interval type cases.

Improvements¶

None.

Bug fixes¶

• Fix Join issues by refactoring qualifiers

• Fix percentile_cont to allow filter and sort order expressions.

• Fix histogram_numeric UDAF.

• Fix the COUNT function when called with multiple args.

New features¶

• Add support for expressions in the GROUP BY clause when the clause is explicitly selected.

• Add error codes to the error messages for better troubleshooting.

Improvements¶

None.

Bug fixes¶

• Fix the window function unsupported cast issue.