Snowflake Container Runtime CPU Version 2.8 (Latest)¶

Added¶

• Stage sharding to cut per-file I/O when reading many small files from a stage. generate_shards() (in snowflake.ml.ray.stage_sharding) repacks many small raw stage files into a few large Arrow IPC (.arrow) shards; read them back with the new SFStageShardDataSource Ray datasource via ray.data.read_datasource, or write shards directly with the lower-level SnowflakeStageShardDatasink.
• Optional file-level disk cache for stage reads. When enabled, SFStageFileSystem caches each stage file on node-local disk on first read (under $TMPDIR/mlrs_disk_cache/file_cache) and serves later reads locally, so repeated passes over the same data (extra epochs, re-runs, and especially many-small-file datasets) skip the network round-trip. It is transparent to every Ray stage datasource (CSV, Parquet, image, text, binary, shard). Off by default; enable cluster-wide with MLRS_FILE_CACHE_ENABLED=true, cap its footprint with MLRS_FILE_CACHE_MAX_DISK_PCT (default 50% of disk; least-recently-used files are evicted above the cap), and point $TMPDIR at node-local NVMe scratch for best results.
• Per-job control of the file cache via a use_file_cache argument on every stage datasource (and create_stage_filesystem): None (default) follows the MLRS_FILE_CACHE_ENABLED flag, True forces the cache on, and False forces it off. The argument is authoritative over the env flag and, unlike an env var, which a worker snapshots at launch, reliably propagates to every Ray worker, giving uniform per-job cache control across a running cluster.
• Extended the file cache to the Snowflake table / DataFrame ingestion path (RayDataIngester). Decoded result batches are cached on node-local disk as Arrow IPC (reusing the same FileCache), so a streamed multi-epoch read serves later epochs from disk instead of re-downloading and re-decoding each result-set chunk from Snowflake. Opt in per job with RayDataIngester(..., use_file_cache=True) (same tri-state semantics as the stage datasources); off by default. Keyed by query_id + batch index, so entries are stable within a job and never stale across queries. Benefits streaming multi-epoch consumers (for example, the PyTorch/generic to_batches() path); it is a no-op for paths that materialize() the dataset, since the Ray object store already caches decoded blocks there.
• HPO now detects uses_snowflake_trainer misconfigurations at trial time. If a distributed Snowflake trainer (XGBEstimator / LightGBMEstimator / PyTorchTrainer) runs inside a trial configured with uses_snowflake_trainer=False, the trial fails fast with an actionable error instead of silently under-allocating resources. Conversely, if a trial declares uses_snowflake_trainer=True but no such trainer runs, it logs a warning suggesting False.

Changed¶

• Training ipywidgets are now enabled by default in vNext notebooks. It was previously gated behind a param _DP_NOTEBOOKS_VNEXT_IPYWIDGETS_DEFAULT_FALSE.
• Breaking: TunerConfig(uses_snowflake_trainer=...) is now required and has no default. Set it to True when the training function trains via a distributed Snowflake trainer (XGBEstimator, LightGBMEstimator, PyTorchTrainer) and False otherwise (for example, an in-process OSS model); omitting it now raises at construction. Previously it defaulted to False, which silently misallocated resources when a distributed trainer was used, wasting roughly half the cluster or starving training. Because the training function is opaque until it runs, this cannot be inferred, so it must be declared.
• HPO/tuning can now run directly on the driver (Ray Task path) instead of being submitted as a Ray Job, enabled via TUNE_DISABLE_RAY_JOB=true. The Ray Job path remains the default for this release; the direct path is being rolled out behind the parameter.
• PyTorch distributed training now runs exclusively on Ray Train. The backend is no longer selectable.
• PyTorch model/checkpoint persistence now uses Ray Train’s native checkpointing. Persist artifacts from your training function via ray.train.report(metrics, checkpoint=ray.train.Checkpoint.from_directory(dir)); Ray Train uploads the checkpoint to the configured stage storage. PyTorchDistributorResult.get_model_dir() and get_checkpoint_location() now return the Ray-managed checkpoint location on the stage (previously the local /tmp/model scratch path).

Deprecated¶

• get_context().get_model_dir(): a local scratch directory that the framework no longer uploads. Persist models via Ray Train’s native checkpointing (ray.train.report(checkpoint=ray.train.Checkpoint.from_directory(dir))).
• get_context().get_artifact_manager(): prefer Ray Train’s native checkpointing. The artifact manager remains available for backward compatibility.

Fixed¶

• Fixed a named-stage artifact_stage_location (for example, DB.SCHEMA.STAGE without a leading @) failing Ray Train storage with invalid URL prefix. get_py_filesystem_for_stage now normalizes named stages to a fully-qualified, @-prefixed URI before they reach Snowflake’s stage APIs.

The following lists the packages available for each Python version of CPU version 2.8.