Head/worker execution for multi-node ML Jobs

In head/worker execution, instance 0 serves as the head node and runs the payload entrypoint once. The remaining instances serve as worker nodes, and the head process distributes work to them. Use this mode when your payload follows this pattern, such as when it uses Ray tasks or Snowflake’s distributed training APIs. This mode is the default and supports function, file, directory, and staged payloads.

Write code for the job

Design your code for distributed processing with Snowflake’s distributed training APIs or Ray.

The following are key patterns and considerations:

Understand node initialization and availability

In multi-node jobs, worker nodes can initialize asynchronously and at different times:

  • Nodes might not all start simultaneously, especially if compute pool resources are limited
  • Some nodes might start seconds or even minutes after others

Before running your payload, ML Jobs waits for the number of instances specified by min_instances, if set; otherwise, it waits for target_instances. The job fails if the required number of instances isn’t available within the timeout period. For more information, see Use for flexible resource management.

You can check available nodes in your job through Ray:

import ray

# Connect to the Ray cluster initialized by ML Jobs.
ray.init(address="auto", ignore_reinit_error=True)
nodes_info = [node for node in ray.nodes() if node["Alive"]]
print(f"Available nodes: {len(nodes_info)}")

Distributed processing patterns

There are multiple patterns you can apply in the payload body of the multi-node job for distributed processing. These patterns use Distributed Modeling Classes and Ray:

Use Snowflake’s distributed training APIs

Snowflake provides optimized trainers for common ML frameworks:

# Inside the ML Job payload body
from snowflake.ml.modeling.distributors.xgboost import XGBEstimator, XGBScalingConfig

# Configure scaling for distributed execution
scaling_config = XGBScalingConfig()

# Create distributed estimator
estimator = XGBEstimator(
    n_estimators=100,
    params={"objective": "reg:squarederror"},
    scaling_config=scaling_config
)

# Train using distributed resources
# NOTE: data_connector and feature_cols excluded for brevity
model = estimator.fit(data_connector, input_cols=feature_cols, label_col="target")

For more information about the available APIs, see Distributed Modeling Classes.

Use native Ray tasks

Another approach is to use Ray’s task-based programming model:

# Inside the ML Job payload body
import ray

# Connect to the Ray cluster initialized by ML Jobs.
ray.init(address="auto", ignore_reinit_error=True)

@ray.remote
def process_chunk(data_chunk):
    # Process a chunk of data
    return processed_result

# Distribute work across available workers
data_chunks = split_data(large_dataset)
futures = [process_chunk.remote(chunk) for chunk in data_chunks]
results = ray.get(futures)

For more information, see Ray’s task programming documentation.

Run the job

You can run multi-node ML Jobs using the same methods as single-node jobs by setting the target_instances parameter. Head/worker execution is the default, so omit parallel or set parallel=False explicitly when you submit the job.

Use the remote decorator

from snowflake.ml.jobs import remote

@remote(
    "MY_COMPUTE_POOL",
    stage_name="payload_stage",
    session=session,
    target_instances=3  # Specify the number of nodes
)
def distributed_training(data_table: str):

    from snowflake.ml.modeling.distributors.xgboost import XGBEstimator, XGBScalingConfig

    # Configure scaling for distributed execution
    scaling_config = XGBScalingConfig()

    # Create distributed estimator
    estimator = XGBEstimator(
        n_estimators=100,
        params={"objective": "reg:squarederror"},
        scaling_config=scaling_config
    )

    # Train using distributed resources
    # NOTE: data_connector and feature_cols excluded for brevity
    model = estimator.fit(data_connector, input_cols=feature_cols, label_col="target")


job = distributed_training("<my_training_data>")

Run a Python file

from snowflake.ml.jobs import submit_file

job = submit_file(
    "<script_path>",
    "MY_COMPUTE_POOL",
    stage_name="<payload_stage>",
    session=session,
    target_instances=<num_training_nodes>  # Specify the number of nodes
)

Run a directory

from snowflake.ml.jobs import submit_directory

job = submit_directory(
    "<script_directory>",
    "MY_COMPUTE_POOL",
    entrypoint="<script_name>",
    stage_name="<payload_stage>",
    session=session,
    target_instances=<num_training_nodes>  # Specify the number of nodes
)

Use min_instances for flexible resource management

For more flexible resource management, you can use the optional min_instances parameter to specify a minimum number of instances required for the job to proceed. If min_instances is set, the job payload runs as soon as the minimum number of nodes becomes available, even if that number is smaller than target_instances.

Use min_instances when you want to:

  • Start training with fewer nodes if the full target isn’t immediately available
  • Reduce wait times when compute pool resources are limited
  • Implement fault-tolerant workflows that can adapt to varying resource availability
from snowflake.ml.jobs import remote

@remote(
    "MY_COMPUTE_POOL",
    stage_name="payload_stage",
    session=session,
    target_instances=5,  # Prefer 5 nodes
    min_instances=3      # But start with at least 3 nodes
)
def flexible_distributed_training(data_table: str):
    import ray

    # Connect to the Ray cluster initialized by ML Jobs.
    ray.init(address="auto", ignore_reinit_error=True)

    # Check how many nodes are available
    available_nodes = sum(1 for node in ray.nodes() if node["Alive"])
    print(f"Training with {available_nodes} nodes")

    # Adapt your training logic based on available resources
    from snowflake.ml.modeling.distributors.xgboost import XGBEstimator, XGBScalingConfig

    scaling_config = XGBScalingConfig(
        num_workers=available_nodes
    )

    estimator = XGBEstimator(
        n_estimators=100,
        params={"objective": "reg:squarederror"},
        scaling_config=scaling_config
    )

    # Train using available distributed resources
    model = estimator.fit(data_connector, input_cols=feature_cols, label_col="target")

job = flexible_distributed_training("<my_training_data>")

Monitor the job

Monitor job status

Job status monitoring is unchanged from single-node jobs. For more information, see Managing ML Jobs.

from snowflake.ml.jobs import MLJob, get_job, list_jobs

# List all jobs
jobs = list_jobs()

# Retrieve an existing job based on ID
job = get_job("<job_id>")  # job is an MLJob instance

# Basic job information
print(f"Job ID: {job.id}")
print(f"Status: {job.status}")  # PENDING, RUNNING, FAILED, DONE, CANCELLING, CANCELLED, INTERNAL_ERROR, DELETED

# Wait for completion
job.wait()

Access logs by node

Use get_logs() or show_logs() to access logs from the head instance or a specific worker instance:

# Get logs from the default (head) instance
logs_default = job.get_logs()

# Get logs from specific instances by ID
logs_instance0 = job.get_logs(instance_id=0)
logs_instance1 = job.get_logs(instance_id=1)
logs_instance2 = job.get_logs(instance_id=2)

# Display logs in the notebook or console
job.show_logs()  # Default (head) instance logs
job.show_logs(instance_id=0)  # Instance 0 (head node) logs

Access the Ray Dashboard

While the job is running, use job.get_ray_dashboard_url() to inspect Ray tasks, actors, nodes, and runtime resources. The Ray Dashboard isn’t available on the CPU_X64_XS instance family. For more information, see Access the Ray Dashboard.

ray_dashboard_url = job.get_ray_dashboard_url()

Common issues and limitations

Use the following information to address common issues that you might encounter.

  • Node connection failures: Worker nodes might fail to connect if the head node finishes and shuts down before workers complete. To prevent this, make the head process wait for all worker results (for example, with ray.get()) before it exits.
  • Memory exhaustion: If jobs fail due to memory issues, use a larger instance family or distribute the workload across more instances.
  • Node availability timeout: If the required number of instances isn’t available before the timeout, the job fails. For more information, see Shared limitations.