Class DataFrameWriter


  • public class DataFrameWriter
    extends Object
    Provides methods for writing data from a DataFrame to supported output destinations. You can write data to the following locations:
    • A Snowflake table
    • A file on a stage

    Saving Data to a Table

    To use this object to write into a table:
    • Access an instance of a DataFrameWriter by calling the DataFrame.write method.
    • Specify the save mode to use (overwrite or append) by calling the mode() method. This method returns a DataFrameWriter that is configured to save data using the specified mode. The default SaveMode is SaveMode.Append.
    • (Optional) If you need to set some options for the save operation (e.g. columnOrder), call the options or option method.
    • Call a saveAsTable method to save the data to the specified destination.

    Saving Data to a File on a Stage

    To use this object to write into a file:
    • Access an instance of a DataFrameWriter by calling the DataFrame.write method.
    • Specify the save mode to use (Overwrite or ErrorIfExists) by calling the DataFrame.mode method. This method returns a DataFrameWriter that is configured to save data using the specified mode. The default SaveMode is SaveMode.ErrorIfExists for this case.
    • (Optional) If you need to set some options for the save operation (e.g. file format options), call the options or option method.
    • Call the method named after a file format to save the data in the specified format:
      • To save the data in CSV format, call the csv method.
      • To save the data in JSON format, call the json method.
      • To save the data in PARQUET format, call the parquet method.
    Since:
    1.1.0
    • Method Detail

      • option

        public DataFrameWriter option​(String key,
                                      Object value)
        Sets the specified option in the DataFrameWriter.

        Sets the specified option for saving data to a table

        Use this method to configure options:

        • columnOrder: save data into a table with table's column name order if saveMode is Append and the target table exists.
        Sets the specified option for saving data to a file on a stage

        Use this method to configure options:

        Note that you cannot use the options or option methods to set the following options:

        • The TYPE format type option.
        • The OVERWRITE copy option. To set this option, use the mode method instead.
          • To set OVERWRITE to TRUE, use SaveMode.Overwrite.
          • To set OVERWRITE to FALSE, use SaveMode.ErrorIfExists.

        For example:

        
         df.write().option("compression", "none").csv("@myStage/prefix");
         
        Parameters:
        key - Name of the option.
        value - Value of the option.
        Returns:
        A DataFrameWriter
        Since:
        1.4.0
      • options

        public DataFrameWriter options​(Map<String,​Object> configs)
        Sets multiple specified options in the DataFrameWriter.

        Sets the specified option for saving data to a table

        Use this method to configure options:

        • columnOrder: save data into a table with table's column name order if saveMode is Append and the target table exists.
        Sets the specified option for saving data to a file on a stage

        Use this method to configure options:

        Note that you cannot use the options or option methods to set the following options:

        • The TYPE format type option.
        • The OVERWRITE copy option. To set this option, use the mode method instead.
          • To set OVERWRITE to TRUE, use SaveMode.Overwrite.
          • To set OVERWRITE to FALSE, use SaveMode.ErrorIfExists.

        For example:

        
         Map<String, Object> configs = new HashMap<>();
         configs.put("compression", "none");
         df.write().options(configs).csv("@myStage/prefix");
         
        Parameters:
        configs - Map of the names of options (e.g. compression, etc.) and their corresponding values.
        Returns:
        A DataFrameWriter
        Since:
        1.5.0
      • csv

        public WriteFileResult csv​(String path)
        Saves the contents of the DataFrame to a CSV file on a stage.

        For example:

        
         df.write().option("compression", "none").csv("@myStage/prefix");
         
        Parameters:
        path - The path (including the stage name) to the CSV file.
        Returns:
        A WriteFileResult
        Since:
        1.5.0
      • json

        public WriteFileResult json​(String path)
        Saves the contents of the DataFrame to a JSON file on a stage.

        NOTE: You can call this method only on a DataFrame that contains a column of the type Variant, Array, or Map. If the DataFrame does not contain a column of one of these types, you must call the to_variant, array_construct, or object_construct to return a DataFrame that contains a column of one of these types.

        For example:

        
         df.write().option("compression", "none").json("@myStage/prefix");
         
        Parameters:
        path - The path (including the stage name) to the JSON file.
        Returns:
        A WriteFileResult
        Since:
        1.5.0
      • parquet

        public WriteFileResult parquet​(String path)
        Saves the contents of the DataFrame to a Parquet file on a stage.

        For example:

        
         df.write().option("compression", "lzo").parquet("@myStage/prefix");
         
        Parameters:
        path - The path (including the stage name) to the Parquet file.
        Returns:
        A WriteFileResult
        Since:
        1.5.0
      • saveAsTable

        public void saveAsTable​(String tableName)
        Writes the data to the specified table in a Snowflake database. tableName can be a fully-qualified object identifier.

        For example:

        
         df.write().saveAsTable("db1.public_schema.table1");
         
        Parameters:
        tableName - Name of the table where the data should be saved.
        Since:
        1.1.0
      • saveAsTable

        public void saveAsTable​(String[] multipartIdentifier)
        Writes the data to the specified table in a Snowflake database.

        For example:

        
         df.write().saveAsTable(new String[]{"db_name", "schema_name", "table_name"})
         
        Parameters:
        multipartIdentifier - A sequence of strings that specify the database name, schema name, and table name.
        Since:
        1.1.0
      • mode

        public DataFrameWriter mode​(String saveMode)
        Returns a new DataFrameWriter with the specified save mode configuration.
        Parameters:
        saveMode - One of the following strings: `"APPEND"`, `"OVERWRITE"`, `"ERRORIFEXISTS"`, or `"IGNORE"`
        Returns:
        This DataFrameWriter
        Since:
        1.1.0
      • mode

        public DataFrameWriter mode​(SaveMode saveMode)
        Returns a new DataFrameWriter with the specified save mode configuration.
        Parameters:
        saveMode - One of the following save modes: SaveMode.Append, SaveMode.Overwrite, SaveMode.ErrorIfExists, SaveMode.Ignore
        Returns:
        This DataFrameWriter
        Since:
        1.1.0
      • async

        public DataFrameWriterAsyncActor async()
        Returns a DataFrameWriterAsyncActor object that can be used to execute DataFrameWriter actions asynchronously.
        Returns:
        A DataFrameWriterAsyncActor object
        Since:
        1.2.0