class DataFrameWriter extends AnyRef

Provides methods for writing data from a DataFrame to supported output destinations.

You can write data to the following locations:

  • A Snowflake table
  • A file on a stage

Saving Data to a Table

To use this object to write into a table:

  1. Access an instance of a DataFrameWriter by calling the DataFrame.write method.
  2. Specify the save mode to use (overwrite or append) by calling the mode method. This method returns a DataFrameWriter that is configured to save data using the specified mode. The default SaveMode is SaveMode.Append .
  3. (Optional) If you need to set some options for the save operation (e.g. columnOrder), call the options or option method.
  4. Call a saveAs* method to save the data to the specified destination.

For example:

df.write.mode("overwrite").saveAsTable("T")

Saving Data to a File on a Stage

To save data to a file on a stage:

  1. Access an instance of a DataFrameWriter by calling the DataFrame.write method.
  2. Specify the save mode to use (Overwrite or ErrorIfExists) by calling the mode method. This method returns a DataFrameWriter that is configured to save data using the specified mode. The default SaveMode is SaveMode.ErrorIfExists for this case.
  3. (Optional) If you need to set some options for the save operation (e.g. file format options), call the options or option method.
  4. Call the method named after a file format to save the data in the specified format:
    • To save the data in CSV format, call the csv method.
    • To save the data in JSON format, call the json method.
    • To save the data in PARQUET format, call the parquet method.

For example:

Example 1: Write a DataFrame to a CSV file.

val result = df.write.csv("@myStage/prefix")

Example 2: Write a DataFrame to a CSV file without compression.

val result = df.write.option("compression", "none").csv("@myStage/prefix")
Since

0.1.0

Linear Supertypes
AnyRef , Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DataFrameWriter
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DataFrameWriter ( dataFrame: DataFrame )

    dataFrame

    Input DataFrame

Value Members

  1. final def != ( arg0: Any ) : Boolean
    Definition Classes
    AnyRef → Any
  2. final def ## () : Int
    Definition Classes
    AnyRef → Any
  3. final def == ( arg0: Any ) : Boolean
    Definition Classes
    AnyRef → Any
  4. def action [ T ] ( funcName: String ) ( func: ⇒ T ) : T
    Attributes
    protected
    Annotations
    @inline ()
  5. final def asInstanceOf [ T0 ] : T0
    Definition Classes
    Any
  6. def async : DataFrameWriterAsyncActor

    Returns a DataFrameWriterAsyncActor object that can be used to execute DataFrameWriter actions asynchronously.

    Returns a DataFrameWriterAsyncActor object that can be used to execute DataFrameWriter actions asynchronously.

    Example:

    val asyncJob = df.write.mode(SaveMode.Overwrite).async.saveAsTable(tableName)
    // At this point, the thread is not blocked. You can perform additional work before
    // calling asyncJob.getResult() to retrieve the results of the action.
    // NOTE: getResult() is a blocking call.
    asyncJob.getResult()
    returns

    A DataFrameWriterAsyncActor object

    Since

    0.11.0

  7. def clone () : AnyRef
    Attributes
    protected[ lang ]
    Definition Classes
    AnyRef
    Annotations
    @throws ( ... ) @native () @HotSpotIntrinsicCandidate ()
  8. def csv ( path: String ) : WriteFileResult

    Saves the contents of the DataFrame to a CSV file on a stage.

    Saves the contents of the DataFrame to a CSV file on a stage.

    Example 1: Write a DataFrame to a CSV file.

    val result = df.write.csv("@myStage/prefix")

    Example 2: Write a DataFrame to a CSV file without compression.

    val result = df.write.option("compression", "none").csv("@myStage/prefix")
    path

    The path (including the stage name) to the CSV file.

    returns

    A WriteFileResult

    Since

    1.5.0

  9. final def eq ( arg0: AnyRef ) : Boolean
    Definition Classes
    AnyRef
  10. def equals ( arg0: Any ) : Boolean
    Definition Classes
    AnyRef → Any
  11. final def getClass () : Class [_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native () @HotSpotIntrinsicCandidate ()
  12. def hashCode () : Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native () @HotSpotIntrinsicCandidate ()
  13. final def isInstanceOf [ T0 ] : Boolean
    Definition Classes
    Any
  14. def json ( path: String ) : WriteFileResult

    Saves the contents of the DataFrame to a JSON file on a stage.

    Saves the contents of the DataFrame to a JSON file on a stage.

    NOTE: You can call this method only on a DataFrame that contains a column of the type Variant, Array, or Map. If the DataFrame does not contain a column of one of these types, you must call the to_variant , array_construct , or object_construct to return a DataFrame that contains a column of one of these types.

    Example 1: Write a DataFrame with one variant to a JSON file.

    val result = session.sql("select to_variant('a')").write.json("@myStage/prefix")

    Example 2: Transform a DataFrame with some columns with array_construct() and write to a JSON file without compression.

    val df = Seq((1, 1.1, "a"), (2, 2.2, "b")).toDF("a", "b", "c")
    val df2 = df.select(array_construct(df.schema.names.map(df(_)): _*))
    val result = df2.write.option("compression", "none").json("@myStage/prefix")

    Example 3: Transform a DataFrame with some columns with object_construct() and write to a JSON file without compression.

    val df = Seq((1, 1.1, "a"), (2, 2.2, "b")).toDF("a", "b", "c")
    val df2 = df.select(object_construct(df.schema.names.map(x => Seq(lit(x), df(x))).flatten: _*))
    val result = df2.write.option("compression", "none").json("@myStage/prefix")
    path

    The path (including the stage name) to the JSON file.

    returns

    A WriteFileResult

    Since

    1.5.0

  15. def mode ( saveMode: SaveMode ) : DataFrameWriter

    Returns a new DataFrameWriter with the specified save mode configuration.

    Returns a new DataFrameWriter with the specified save mode configuration.

    saveMode

    One of the following save modes: SaveMode.Append , SaveMode.Overwrite , SaveMode.ErrorIfExists , SaveMode.Ignore

    Since

    0.1.0

  16. def mode ( saveMode: String ) : DataFrameWriter

    Returns a new DataFrameWriter with the specified save mode configuration.

    Returns a new DataFrameWriter with the specified save mode configuration.

    saveMode

    One of the following strings: "APPEND" , "OVERWRITE" , "ERRORIFEXISTS" , or "IGNORE"

    Since

    0.1.0

  17. final def ne ( arg0: AnyRef ) : Boolean
    Definition Classes
    AnyRef
  18. final def notify () : Unit
    Definition Classes
    AnyRef
    Annotations
    @native () @HotSpotIntrinsicCandidate ()
  19. final def notifyAll () : Unit
    Definition Classes
    AnyRef
    Annotations
    @native () @HotSpotIntrinsicCandidate ()
  20. def option ( key: String , value: Any ) : DataFrameWriter

    Sets the specified option in the DataFrameWriter.

    Sets the specified option in the DataFrameWriter.

    Sets the specified option for saving data to a table

    Use this method to configure options:

    • columnOrder: save data into a table with table's column name order if saveMode is Append and target table exists.

    Sets the specified option for saving data to a file on a stage

    Use this method to configure options:

    Note that you cannot use the option and options methods to set the following options:

    • The TYPE format type option.
    • The OVERWRITE copy option. To set this option, use the mode method instead.
      • To set OVERWRITE to TRUE , use SaveMode.Overwrite .
      • To set OVERWRITE to FALSE , use SaveMode.ErrorIfExists .

    Example 1: Write a DataFrame to a CSV file.

    val result = df.write.csv("@myStage/prefix")

    Example 2: Write a DataFrame to a CSV file without compression.

    val result = df.write.option("compression", "none").csv("@myStage/prefix")
    key

    Name of the option.

    value

    Value of the option.

    returns

    A DataFrameWriter

    Since

    1.4.0

  21. def options ( configs: Map [ String , Any ] ) : DataFrameWriter

    Sets multiple specified options in the DataFrameWriter.

    Sets multiple specified options in the DataFrameWriter.

    Sets the specified options for saving Data to a Table

    Use this method to configure options:

    • columnOrder: save data into a table with table's column name order if saveMode is Append and target table exists.

    Sets the specified options for saving data to a file on a stage

    Use this method to configure options:

    Note that you cannot use the option and options methods to set the following options:

    • The TYPE format type option.
    • The OVERWRITE copy option. To set this option, use the mode method instead.
      • To set OVERWRITE to TRUE , use SaveMode.Overwrite .
      • To set OVERWRITE to FALSE , use SaveMode.ErrorIfExists .

    Example 1: Write a DataFrame to a CSV file.

    val result = df.write.csv("@myStage/prefix")

    Example 2: Write a DataFrame to a CSV file without compression.

    val result = df.write.option("compression", "none").csv("@myStage/prefix")
    configs

    Map of the names of options (e.g. compression , etc.) and their corresponding values.

    returns

    A DataFrameWriter

    Since

    1.5.0

  22. def parquet ( path: String ) : WriteFileResult

    Saves the contents of the DataFrame to a Parquet file on a stage.

    Saves the contents of the DataFrame to a Parquet file on a stage.

    Example 1: Write a DataFrame to a parquet file.

    val result = df.write.parquet("@myStage/prefix")

    Example 2: Write a DataFrame to a Parquet file without compression.

    val result = df.write.option("compression", "LZO").parquet("@myStage/prefix")
    path

    The path (including the stage name) to the Parquet file.

    returns

    A WriteFileResult

    Since

    1.5.0

  23. def saveAsTable ( multipartIdentifier: List [ String ] ) : Unit

    Writes the data to the specified table in a Snowflake database.

    Writes the data to the specified table in a Snowflake database.

    For example:

      val list = new java.util.ArrayList[String](3)
      list.add(db)
      list.add(sc)
      list.add(tableName)
    df.write.saveAsTable(list)
    multipartIdentifier

    A list of strings that specify the database name, schema name, and table name.

    Since

    0.5.0

  24. def saveAsTable ( multipartIdentifier: Seq [ String ] ) : Unit

    Writes the data to the specified table in a Snowflake database.

    Writes the data to the specified table in a Snowflake database.

    For example:

    df.write.saveAsTable(Seq("db_name", "schema_name", "table_name"))
    multipartIdentifier

    A sequence of strings that specify the database name, schema name, and table name (e.g. Seq("database_name", "schema_name", "table_name") ).

    Since

    0.5.0

  25. def saveAsTable ( tableName: String ) : Unit

    Writes the data to the specified table in a Snowflake database.

    Writes the data to the specified table in a Snowflake database. tableName can be a fully-qualified object identifier.

    For example:

    df.write.saveAsTable("db1.public_schema.table1")
    tableName

    Name of the table where the data should be saved.

    Since

    0.1.0

  26. final def synchronized [ T0 ] ( arg0: ⇒ T0 ) : T0
    Definition Classes
    AnyRef
  27. def toString () : String
    Definition Classes
    AnyRef → Any
  28. final def wait ( arg0: Long , arg1: Int ) : Unit
    Definition Classes
    AnyRef
    Annotations
    @throws ( ... )
  29. final def wait ( arg0: Long ) : Unit
    Definition Classes
    AnyRef
    Annotations
    @throws ( ... ) @native ()
  30. final def wait () : Unit
    Definition Classes
    AnyRef
    Annotations
    @throws ( ... )

Deprecated Value Members

  1. def finalize () : Unit
    Attributes
    protected[ lang ]
    Definition Classes
    AnyRef
    Annotations
    @throws ( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from AnyRef

Inherited from Any

Ungrouped