Aggregate the data in the DataFrame. Use this method if you don't need to group the data ( groupBy ).

For the input value, pass in expressions that apply aggregation functions to columns (functions that are defined in the functions object).

The following example calculates the maximum value of the num_sales column and the mean value of the price column:

For example:

import com.snowflake.snowpark.functions._

val dfAgg = df.agg(Array(max($"num_sales"), mean($"price")))

Aggregate the data in the DataFrame. Use this method if you don't need to group the data ( groupBy ).

For the input value, pass in expressions that apply aggregation functions to columns (functions that are defined in the functions object).

The following example calculates the maximum value of the num_sales column and the mean value of the price column:

import com.snowflake.snowpark.functions._

val dfAgg = df.agg(Seq(max($"num_sales"), mean($"price")))

Aggregate the data in the DataFrame. Use this method if you don't need to group the data ( groupBy ).

For the input value, pass in expressions that apply aggregation functions to columns (functions that are defined in the functions object).

The following example calculates the maximum value of the num_sales column and the mean value of the price column:

For example:

import com.snowflake.snowpark.functions._

val dfAgg = df.agg(max($"num_sales"), mean($"price"))

Aggregate the data in the DataFrame. Use this method if you don't need to group the data ( groupBy ).

For the input, pass in a Map that specifies the column names and aggregation functions. For each pair in the Map:

• Set the key to the name of the column to aggregate.
• Set the value to the name of the aggregation function to use on that column.

The following example calculates the maximum value of the num_sales column and the average value of the price column:

val dfAgg = df.agg(Seq("num_sales" -> "max", "price" -> "mean"))

This is equivalent to calling agg after calling groupBy without a column name:

val dfAgg = df.groupBy().agg(Seq(df("num_sales") -> "max", df("price") -> "mean"))

Aggregate the data in the DataFrame. Use this method if you don't need to group the data ( groupBy ).

For the input, pass in a Map that specifies the column names and aggregation functions. For each pair in the Map:

• Set the key to the name of the column to aggregate.
• Set the value to the name of the aggregation function to use on that column.

The following example calculates the maximum value of the num_sales column and the average value of the price column:

val dfAgg = df.agg("num_sales" -> "max", "price" -> "mean")

This is equivalent to calling agg after calling groupBy without a column name:

val dfAgg = df.groupBy().agg(df("num_sales") -> "max", df("price") -> "mean")

Returns the current DataFrame aliased as the input alias name.

For example:

val df2 = df.alias("A")
df2.select(df2.col("A.num"))

Returns a reference to a column in the DataFrame. This method is identical to DataFrame.col .

Returns a DataFrameAsyncActor object that can be used to execute DataFrame actions asynchronously.

Example:

val asyncJob = df.async.collect()
// At this point, the thread is not blocked. You can perform additional work before
// calling asyncJob.getResult() to retrieve the results of the action.
// NOTE: getResult() is a blocking call.
val rows = asyncJob.getResult()

Caches the content of this DataFrame to create a new cached DataFrame.

All subsequent operations on the returned cached DataFrame are performed on the cached data and have no effect on the original DataFrame.

Returns a clone of this DataFrame.

Returns a reference to a column in the DataFrame.

Executes the query representing this DataFrame and returns the result as an Array of Row objects.

Executes the query representing this DataFrame and returns the number of rows in the result (similar to the COUNT function in SQL).

Creates a temporary view that returns the same results as this DataFrame.

You can use the view in subsequent SQL queries and statements during the current session. The temporary view is only available in the session in which it is created.

In multipartIdentifer , you can include the database and schema name to specify a fully-qualified name. If no database name or schema name are specified, the view will be created in the current database or schema.

The view name must be a valid Snowflake identifier .

Creates a temporary view that returns the same results as this DataFrame.

You can use the view in subsequent SQL queries and statements during the current session. The temporary view is only available in the session in which it is created.

In multipartIdentifer , you can include the database and schema name to specify a fully-qualified name. If no database name or schema name are specified, the view will be created in the current database or schema.

The view name must be a valid Snowflake identifier .

Creates a temporary view that returns the same results as this DataFrame.

You can use the view in subsequent SQL queries and statements during the current session. The temporary view is only available in the session in which it is created.

For viewName , you can include the database and schema name (i.e. specify a fully-qualified name). If no database name or schema name are specified, the view will be created in the current database or schema.

viewName must be a valid Snowflake identifier .

Creates a view that captures the computation expressed by this DataFrame.

In multipartIdentifer , you can include the database and schema name to specify a fully-qualified name. If no database name or schema name are specified, the view will be created in the current database or schema.

The view name must be a valid Snowflake identifier .

Creates a view that captures the computation expressed by this DataFrame.

In multipartIdentifer , you can include the database and schema name to specify a fully-qualified name. If no database name or schema name are specified, the view will be created in the current database or schema.

The view name must be a valid Snowflake identifier .

Creates a view that captures the computation expressed by this DataFrame.

For viewName , you can include the database and schema name (i.e. specify a fully-qualified name). If no database name or schema name are specified, the view will be created in the current database or schema.

viewName must be a valid Snowflake identifier .

Performs a cross join, which returns the cartesian product of the current DataFrame and another DataFrame ( right ).

If the current and right DataFrames have columns with the same name, and you need to refer to one of these columns in the returned DataFrame, use the apply or col function on the current or right DataFrame to disambiguate references to these columns.

For example:

val dfCrossJoin = left.crossJoin(right)
val project = dfCrossJoin.select(left("common_col") + right("common_col"))

Performs an SQL GROUP BY CUBE

Performs an SQL GROUP BY CUBE on the DataFrame.

Performs an SQL GROUP BY CUBE on the DataFrame.

Performs an SQL GROUP BY CUBE on the DataFrame.

Performs an SQL GROUP BY CUBE on the DataFrame.

Returns a new DataFrame that contains only the rows with distinct values from the current DataFrame.

This is equivalent to performing a SELECT DISTINCT in SQL.

Returns a new DataFrame that excludes the specified column expressions from the output.

This is functionally equivalent to calling select and passing in all columns except the ones to exclude.

This method throws a SnowparkClientException if:

• A specified column does not have a name, or
• The resulting DataFrame has no output columns.

Returns a new DataFrame that excludes the specified column expressions from the output.

This is functionally equivalent to calling select and passing in all columns except the ones to exclude.

This method throws a SnowparkClientException if:

• A specified column does not have a name, or
• The resulting DataFrame has no output columns.

Returns a new DataFrame that excludes the columns specified by the expressions from the output.

This is functionally equivalent to calling select and passing in all columns except the ones to exclude.

This method throws a SnowparkClientException if:

• A specified column does not have a name, or
• The resulting DataFrame has no output columns.

Returns a new DataFrame that excludes the columns with the specified names from the output.

This is functionally equivalent to calling select and passing in all columns except the ones to exclude.

Throws SnowparkClientException if the resulting DataFrame contains no output columns.

Returns a new DataFrame that excludes the columns with the specified names from the output.

This is functionally equivalent to calling select and passing in all columns except the ones to exclude.

Throws SnowparkClientException if the resulting DataFrame contains no output columns.

Returns a new DataFrame that excludes the columns with the specified names from the output.

This is functionally equivalent to calling select and passing in all columns except the ones to exclude.

Throws SnowparkClientException if the resulting DataFrame contains no output columns.

Creates a new DataFrame by removing duplicated rows on given subset of columns. If no subset of columns specified, this function is same as distinct() function. The result is non-deterministic when removing duplicated rows from the subset of columns but not all columns. For example: Supposes we have a DataFrame df , which contains three rows (a, b, c): (1, 1, 1), (1, 1, 2), (1, 2, 3) The result of df.dropDuplicates("a", "b") can be either (1, 1, 1), (1, 2, 3) or (1, 1, 2), (1, 2, 3)

Returns a new DataFrame that contains all the rows from the current DataFrame except for the rows that also appear in another DataFrame ( other ). Duplicate rows are eliminated.

For example:

val df1except2 = df1.except(df2)

Prints the list of queries that will be executed to evaluate this DataFrame. Prints the query execution plan if only one SELECT/DML/DDL statement will be executed.

For more information about the query execution plan, see the EXPLAIN command.

Filters rows based on the specified conditional expression (similar to WHERE in SQL).

For example:

val dfFiltered = df.filter($"colA" > 1 && $"colB" < 100)

Executes the query representing this DataFrame and returns the first n rows of the results.

Executes the query representing this DataFrame and returns the first row of results.

Flattens (explodes) compound values into multiple rows (similar to the SQL FLATTEN function).

The flatten method adds the following columns to the returned DataFrame:

• SEQ
• KEY
• PATH
• INDEX
• VALUE
• THIS

If this DataFrame also has columns with the names above, you can disambiguate the columns by using the this("value") syntax.

For example, if the current DataFrame has a column named value :

val table1 = session.sql(
  "select parse_json(value) as value from values('[1,2]') as T(value)")
val flattened = table1.flatten(table1("value"), "", outer = false,
  recursive = false, "both")
flattened.select(table1("value"), flattened("value").as("newValue")).show()

Flattens (explodes) compound values into multiple rows (similar to the SQL FLATTEN function).

The flatten method adds the following columns to the returned DataFrame:

• SEQ
• KEY
• PATH
• INDEX
• VALUE
• THIS

If this DataFrame also has columns with the names above, you can disambiguate the columns by using the this("value") syntax.

For example, if the current DataFrame has a column named value :

val table1 = session.sql(
  "select parse_json(value) as value from values('[1,2]') as T(value)")
val flattened = table1.flatten(table1("value"))
flattened.select(table1("value"), flattened("value").as("newValue")).show()

Groups rows by the columns specified by name (similar to GROUP BY in SQL).

This method returns a RelationalGroupedDataFrame that you can use to perform aggregations on each group of data.

Groups rows by the columns specified by name (similar to GROUP BY in SQL).

This method returns a RelationalGroupedDataFrame that you can use to perform aggregations on each group of data.

Groups rows by the columns specified by name (similar to GROUP BY in SQL).

This method returns a RelationalGroupedDataFrame that you can use to perform aggregations on each group of data.

Groups rows by the columns specified by expressions (similar to GROUP BY in SQL).

This method returns a RelationalGroupedDataFrame that you can use to perform aggregations on each group of data.

Groups rows by the columns specified by expressions (similar to GROUP BY in SQL).

This method returns a RelationalGroupedDataFrame that you can use to perform aggregations on each group of data.

Returns a RelationalGroupedDataFrame that you can use to perform aggregations on the underlying DataFrame.

Groups rows by the columns specified by expressions (similar to GROUP BY in SQL).

This method returns a RelationalGroupedDataFrame that you can use to perform aggregations on each group of data.

Performs an SQL GROUP BY GROUPING SETS on the DataFrame.

GROUP BY GROUPING SETS is an extension of the GROUP BY clause that allows computing multiple group-by clauses in a single statement. The group set is a set of dimension columns.

GROUP BY GROUPING SETS is equivalent to the UNION of two or more GROUP BY operations in the same result set:

df.groupByGroupingSets(GroupingSets(Set(col("a")))) is equivalent to df.groupBy("a")

and

df.groupByGroupingSets(GroupingSets(Set(col("a")), Set(col("b")))) is equivalent to df.groupBy("a") union df.groupBy("b")

Performs an SQL GROUP BY GROUPING SETS on the DataFrame.

GROUP BY GROUPING SETS is an extension of the GROUP BY clause that allows computing multiple GROUP BY clauses in a single statement. The group set is a set of dimension columns.

GROUP BY GROUPING SETS is equivalent to the UNION of two or more GROUP BY operations in the same result set:

df.groupByGroupingSets(GroupingSets(Set(col("a")))) is equivalent to df.groupBy("a")

and

df.groupByGroupingSets(GroupingSets(Set(col("a")), Set(col("b")))) is equivalent to df.groupBy("a") union df.groupBy("b")

Returns a new DataFrame that contains the intersection of rows from the current DataFrame and another DataFrame ( other ). Duplicate rows are eliminated.

For example:

val dfIntersectionOf1and2 = df1.intersect(df2)

Joins the current DataFrame with the output of the specified user-defined table function (UDTF) func .

To specify a PARTITION BY or ORDER BY clause, use the partitionBy and orderBy arguments.

For example:

val tf = session.udtf.registerTemporary(TableFunc1)
df.join(tf(Map("arg1" -> df("col1")),Seq(df("col2")), Seq(df("col1"))))

Joins the current DataFrame with the output of the specified table function func .

For example:

// The following example uses the flatten function to explode compound values from
// column 'a' in this DataFrame into multiple columns.

import com.snowflake.snowpark.functions._
import com.snowflake.snowpark.tableFunctions._

df.join(
  tableFunctions.flatten(parse_json(df("a")))
)

Joins the current DataFrame with the output of the specified user-defined table function (UDTF) func .

To pass arguments to the table function, use the args argument of this method. Pass in a Map of parameter names and values. In these values, you can include references to columns in this DataFrame.

To specify a PARTITION BY or ORDER BY clause, use the partitionBy and orderBy arguments.

For example:

// The following example passes the values in the column `col1` to the
// user-defined tabular function (UDTF) `udtf`, partitioning the
// data by `col2` and sorting the data by `col1`. The example returns
// a new DataFrame that joins the contents of the current DataFrame with
// the output of the UDTF.
df.join(
  tableFunction("udtf"),
  Map("arg1" -> df("col1"),
  Seq(df("col2")), Seq(df("col1")))
)

Joins the current DataFrame with the output of the specified table function func that takes named parameters (e.g. flatten ).

To pass arguments to the table function, use the args argument of this method. Pass in a Map of parameter names and values. In these values, you can include references to columns in this DataFrame.

For example:

// The following example uses the flatten function to explode compound values from
// column 'a' in this DataFrame into multiple columns.

import com.snowflake.snowpark.functions._
import com.snowflake.snowpark.tableFunctions._

df.join(
  tableFunction("flatten"),
  Map("input" -> parse_json(df("a")))
)

Joins the current DataFrame with the output of the specified user-defined table function (UDTF) func .

To pass arguments to the table function, use the args argument of this method. In the table function arguments, you can include references to columns in this DataFrame.

To specify a PARTITION BY or ORDER BY clause, use the partitionBy and orderBy arguments.

For example:

// The following example passes the values in the column `col1` to the
// user-defined tabular function (UDTF) `udtf`, partitioning the
// data by `col2` and sorting the data by `col1`. The example returns
// a new DataFrame that joins the contents of the current DataFrame with
// the output of the UDTF.
df.join(TableFunction("udtf"), Seq(df("col1")), Seq(df("col2")), Seq(df("col1")))

Joins the current DataFrame with the output of the specified table function func .

To pass arguments to the table function, use the args argument of this method. In the table function arguments, you can include references to columns in this DataFrame.

For example:

// The following example uses the split_to_table function to split
// column 'a' in this DataFrame on the character ','.
// Each row in this DataFrame will produce N rows in the resulting DataFrame,
// where N is the number of tokens in the column 'a'.
import com.snowflake.snowpark.functions._
import com.snowflake.snowpark.tableFunctions._

df.join(split_to_table, Seq(df("a"), lit(",")))

Joins the current DataFrame with the output of the specified table function func .

To pass arguments to the table function, use the firstArg and remaining arguments of this method. In the table function arguments, you can include references to columns in this DataFrame.

For example:

// The following example uses the split_to_table function to split
// column 'a' in this DataFrame on the character ','.
// Each row in the current DataFrame will produce N rows in the resulting DataFrame,
// where N is the number of tokens in the column 'a'.

import com.snowflake.snowpark.functions._
import com.snowflake.snowpark.tableFunctions._

df.join(split_to_table, df("a"), lit(","))

Performs a join of the specified type ( joinType ) with the current DataFrame and another DataFrame ( right ) using the join condition specified in an expression ( joinExpr ).

To disambiguate columns with the same name in the left DataFrame and right DataFrame, use the apply or col method of each DataFrame ( df("col") or df.col("col") ). You can use this approach to disambiguate columns in the joinExprs parameter and to refer to columns in the returned DataFrame.

For example:

val dfJoin = df1.join(df2, df1("a") === df2("b"), "left")
val dfJoin2 = df1.join(df2, df1("a") === df2("b") && df1("c" === df2("d"), "outer")
val dfJoin3 = df1.join(df2, df1("a") === df2("a") && df1("b" === df2("b"), "outer")
// If both df1 and df2 contain column 'c'
val project = dfJoin3.select(df1("c") + df2("c"))

If you need to join a DataFrame with itself, keep in mind that there is no way to distinguish between columns on the left and right sides in a join expression. For example:

val dfJoined = df.join(df, df("a") === df("b"), joinType) // Column references are ambiguous

To do a self-join, you can you either clone( clone ) the DataFrame as follows,

val clonedDf = df.clone
val dfJoined = df.join(clonedDf, df("a") === clonedDf("b"), joinType)

or you can call a join method that allows you to pass in 'usingColumns' parameter.

Performs a default inner join of the current DataFrame and another DataFrame ( right ) using the join condition specified in an expression ( joinExpr ).

To disambiguate columns with the same name in the left DataFrame and right DataFrame, use the apply or col method of each DataFrame ( df("col") or df.col("col") ). You can use this approach to disambiguate columns in the joinExprs parameter and to refer to columns in the returned DataFrame.

For example:

val dfJoin = df1.join(df2, df1("a") === df2("b"))
val dfJoin2 = df1.join(df2, df1("a") === df2("b") && df1("c" === df2("d"))
val dfJoin3 = df1.join(df2, df1("a") === df2("a") && df1("b" === df2("b"))
// If both df1 and df2 contain column 'c'
val project = dfJoin3.select(df1("c") + df2("c"))

If you need to join a DataFrame with itself, keep in mind that there is no way to distinguish between columns on the left and right sides in a join expression. For example:

val dfJoined = df.join(df, df("a") === df("b")) // Column references are ambiguous

As a workaround, you can either construct the left and right DataFrames separately, or you can call a join method that allows you to pass in 'usingColumns' parameter.

Performs a join of the specified type ( joinType ) with the current DataFrame and another DataFrame ( right ) on a list of columns ( usingColumns ).

The method assumes that the columns in usingColumns have the same meaning in the left and right DataFrames.

For example:

val dfLeftJoin = df1.join(df2, Seq("a"), "left")
val dfOuterJoin = df1.join(df2, Seq("a", "b"), "outer")

Performs a default inner join of the current DataFrame and another DataFrame ( right ) on a list of columns ( usingColumns ).

The method assumes that the columns in usingColumns have the same meaning in the left and right DataFrames.

For example:

val dfJoinOnColA = df.join(df2, Seq("a"))
val dfJoinOnColAAndColB = df.join(df2, Seq("a", "b"))

Performs a default inner join of the current DataFrame and another DataFrame ( right ) on a column ( usingColumn ).

The method assumes that the usingColumn column has the same meaning in the left and right DataFrames.

For example:

val result = left.join(right, "a")

Performs a default inner join of the current DataFrame and another DataFrame ( right ).

Because this method does not specify a join condition, the returned DataFrame is a cartesian product of the two DataFrames.

If the current and right DataFrames have columns with the same name, and you need to refer to one of these columns in the returned DataFrame, use the apply or col function on the current or right DataFrame to disambiguate references to these columns.

For example:

val result = left.join(right)
val project = result.select(left("common_col") + right("common_col"))

Returns a new DataFrame that contains at most n rows from the current DataFrame (similar to LIMIT in SQL).

Note that this is a transformation method and not an action method.

Returns a DataFrameNaFunctions object that provides functions for handling missing values in the DataFrame.

Performs a natural join of the specified type ( joinType ) with the current DataFrame and another DataFrame ( right ).

For example:

val dfNaturalJoin = df.naturalJoin(df2, "left")

Performs a natural join (a default inner join) of the current DataFrame and another DataFrame ( right ).

For example:

val dfNaturalJoin = df.naturalJoin(df2)

Note that this is equivalent to:

val dfNaturalJoin = df.naturalJoin(df2, "inner")

Rotates this DataFrame by turning the unique values from one column in the input expression into multiple columns and aggregating results where required on any remaining column values.

Only one aggregate is supported with pivot.

For example:

val dfPivoted = df.pivot(col("col_1"), Seq(1,2,3)).agg(sum(col("col_2")))

Rotates this DataFrame by turning the unique values from one column in the input expression into multiple columns and aggregating results where required on any remaining column values.

Only one aggregate is supported with pivot.

For example:

val dfPivoted = df.pivot("col_1", Seq(1,2,3)).agg(sum(col("col_2")))

Randomly splits the current DataFrame into separate DataFrames, using the specified weights.

NOTE:

• If only one weight is specified, the returned DataFrame array only includes the current DataFrame.
• If multiple weights are specified, the current DataFrame will be cached before being split.

Returns a DataFrame with the specified column col renamed as newName .

This example renames the column A as NEW_A in the DataFrame.

val df = session.sql("select 1 as A, 2 as B")
val dfRenamed = df.rename("NEW_A", col("A"))

Performs an SQL GROUP BY ROLLUP on the DataFrame.

Performs an SQL GROUP BY ROLLUP on the DataFrame.

Performs an SQL GROUP BY ROLLUP on the DataFrame.

Performs an SQL GROUP BY ROLLUP on the DataFrame.

Performs an SQL GROUP BY ROLLUP on the DataFrame.

Performs an SQL GROUP BY ROLLUP on the DataFrame.

Returns a new DataFrame that contains a sampling of rows from the current DataFrame.

NOTE:

• The number of rows returned may be close to (but not exactly equal to) (probabilityFraction * totalRowCount) .
• The Snowflake SAMPLE function supports specifying 'probability' as a percentage number. The range of 'probability' is [0.0, 100.0] . The conversion formula is probability = probabilityFraction * 100 .

Returns a new DataFrame with a sample of N rows from the underlying DataFrame.

NOTE:

• If the row count in the DataFrame is larger than the requested number of rows, the method returns a DataFrame containing the number of requested rows.
• If the row count in the DataFrame is smaller than the requested number of rows, the method returns a DataFrame containing all rows.

Returns the definition of the columns in this DataFrame (the "relational schema" for the DataFrame).

Returns a new DataFrame with a subset of named columns (similar to SELECT in SQL).

For example:

val dfSelected = df.select(Array("col1", "col2"))

Returns a new DataFrame with a subset of named columns (similar to SELECT in SQL).

For example:

val dfSelected = df.select(Seq("col1", "col2", "col3"))

Returns a new DataFrame with a subset of named columns (similar to SELECT in SQL).

For example:

val dfSelected = df.select("col1", "col2", "col3")

Returns a new DataFrame with the specified Column expressions as output (similar to SELECT in SQL). Only the Columns specified as arguments will be present in the resulting DataFrame.

You can use any Column expression.

For example:

val dfSelected =
  df.select(Array(df.col("col1"), lit("abc"), df.col("col1") + df.col("col2")))

Returns a new DataFrame with the specified Column expressions as output (similar to SELECT in SQL). Only the Columns specified as arguments will be present in the resulting DataFrame.

You can use any Column expression.

For example:

val dfSelected = df.select(Seq($"col1", substring($"col2", 0, 10), df("col3") + df("col4")))

Returns a new DataFrame with the specified Column expressions as output (similar to SELECT in SQL). Only the Columns specified as arguments will be present in the resulting DataFrame.

You can use any Column expression.

For example:

val dfSelected = df.select($"col1", substring($"col2", 0, 10), df("col3") + df("col4"))

Evaluates this DataFrame and prints out the first n rows with the specified maximum number of characters per column.

Evaluates this DataFrame and prints out the first n rows.

Evaluates this DataFrame and prints out the first ten rows.

Sorts a DataFrame by the specified expressions (similar to ORDER BY in SQL).

For example:

val dfSorted = df.sort(Array(col("col1").asc, col("col2").desc, col("col3")))

Sorts a DataFrame by the specified expressions (similar to ORDER BY in SQL).

For example:

val dfSorted = df.sort(Seq($"colA", $"colB".desc))

Sorts a DataFrame by the specified expressions (similar to ORDER BY in SQL).

For example:

val dfSorted = df.sort($"colA", $"colB".asc)

Returns a DataFrameStatFunctions object that provides statistic functions.

Creates a new DataFrame containing the data in the current DataFrame but in columns with the specified names.

You can use this method to assign column names when constructing a DataFrame. For example:

For example:

val df = session.createDataFrame(Seq((1, "a"))).toDF(Array("a", "b"))

This returns a DataFrame containing the following:

-------------
|"A"  |"B"  |
-------------
|1    |2    |
|3    |4    |
-------------

If you imported <session_var>.implicits._ , you can use the following syntax to create the DataFrame from a Seq and call toDF to assign column names to the returned DataFrame:

import mysession.implicits_
var df = Seq((1, 2), (3, 4)).toDF(Array("a", "b"))

The number of column names that you pass in must match the number of columns in the current DataFrame.

Creates a new DataFrame containing the data in the current DataFrame but in columns with the specified names.

You can use this method to assign column names when constructing a DataFrame. For example:

For example:

var df = session.createDataFrame(Seq((1, 2), (3, 4))).toDF(Seq("a", "b"))

This returns a DataFrame containing the following:

-------------
|"A"  |"B"  |
-------------
|1    |2    |
|3    |4    |
-------------

If you imported <session_var>.implicits._ , you can use the following syntax to create the DataFrame from a Seq and call toDF to assign column names to the returned DataFrame:

import mysession.implicits_
var df = Seq((1, 2), (3, 4)).toDF(Seq("a", "b"))

The number of column names that you pass in must match the number of columns in the current DataFrame.

Creates a new DataFrame containing the columns with the specified names.

You can use this method to assign column names when constructing a DataFrame. For example:

For example:

var df = session.createDataFrame(Seq((1, "a")).toDF(Seq("a", "b"))

This returns a DataFrame containing the following:

-------------
|"A"  |"B"  |
-------------
|1    |2    |
|3    |4    |
-------------

if you imported <session_var>.implicits._ , you can use the following syntax to create the DataFrame from a Seq and call toDF to assign column names to the returned DataFrame:

import mysession.implicits_
var df = Seq((1, 2), (3, 4)).toDF(Seq("a", "b"))

The number of column names that you pass in must match the number of columns in the current DataFrame.

Executes the query representing this DataFrame and returns an iterator of Row objects that you can use to retrieve the results.

Unlike the collect method, this method does not load all data into memory at once.

Returns a new DataFrame that contains all the rows in the current DataFrame and another DataFrame ( other ), excluding any duplicate rows. Both input DataFrames must contain the same number of columns.

For example:

val df1and2 = df1.union(df2)

Returns a new DataFrame that contains all the rows in the current DataFrame and another DataFrame ( other ), including any duplicate rows. Both input DataFrames must contain the same number of columns.

For example:

val df1and2 = df1.unionAll(df2)

Returns a new DataFrame that contains all the rows in the current DataFrame and another DataFrame ( other ), including any duplicate rows.

This method matches the columns in the two DataFrames by their names, not by their positions. The columns in the other DataFrame are rearranged to match the order of columns in the current DataFrame.

For example:

val df1and2 = df1.unionAllByName(df2)

Returns a new DataFrame that contains all the rows in the current DataFrame and another DataFrame ( other ), excluding any duplicate rows.

This method matches the columns in the two DataFrames by their names, not by their positions. The columns in the other DataFrame are rearranged to match the order of columns in the current DataFrame.

For example:

val df1and2 = df1.unionByName(df2)

Filters rows based on the specified conditional expression (similar to WHERE in SQL). This is equivalent to calling filter .

For example:

// The following two result in the same SQL query:
pricesDF.filter($"price" > 100)
pricesDF.where($"price" > 100)

Returns a DataFrame with an additional column with the specified name ( colName ). The column is computed by using the specified expression ( col ).

If a column with the same name already exists in the DataFrame, that column is replaced by the new column.

This example adds a new column named mean_price that contains the mean of the existing price column in the DataFrame.

val dfWithMeanPriceCol = df.withColumn("mean_price", mean($"price"))

Returns a DataFrame with additional columns with the specified names ( colNames ). The columns are computed by using the specified expressions ( cols ).

If columns with the same names already exist in the DataFrame, those columns are replaced by the new columns.

This example adds new columns named mean_price and avg_price that contain the mean and average of the existing price column.

val dfWithAddedColumns = df.withColumn(
    Seq("mean_price", "avg_price"), Seq(mean($"price"), avg($"price") )

Returns a DataFrameWriter object that you can use to write the data in the DataFrame to any supported destination. The Default SaveMode for the returned DataFrameWriter is Append .

Example:

df.write.saveAsTable("table1")