Snowpark 1.13.1 - com.snowflake.snowpark.functions

final def != ( arg0: Any ) : Boolean

Definition Classes: AnyRef → Any

final def ## () : Int

Definition Classes: AnyRef → Any

final def == ( arg0: Any ) : Boolean

Definition Classes: AnyRef → Any

def abs ( e: Column ) : Column

Returns the absolute value of a numeric expression.

Since: 0.1.0

def acos ( e: Column ) : Column

Computes the inverse cosine (arc cosine) of its input; the result is a number in the interval [-pi, pi].

Since: 0.1.0

def add_months ( startDate: Column , numMonths: Column ) : Column

Adds or subtracts a specified number of months to a date or timestamp, preserving the end-of-month information.

Since: 0.1.0

def any_value ( e: Column ) : Column

Returns a non-deterministic value for the specified column.

Since: 0.12.0

def approx_count_distinct ( e: Column ) : Column

Uses HyperLogLog to return an approximation of the distinct cardinality of the input (i.e.

Uses HyperLogLog to return an approximation of the distinct cardinality of the input (i.e. returns an approximation of COUNT(DISTINCT col) ).

Since: 0.1.0

def approx_percentile ( col: Column , percentile: Double ) : Column

Returns an approximated value for the desired percentile.

Returns an approximated value for the desired percentile. This function uses the t-Digest algorithm.

Since: 0.2.0

def approx_percentile_accumulate ( col: Column ) : Column

Returns the internal representation of the t-Digest state (as a JSON object) at the end of aggregation.

Returns the internal representation of the t-Digest state (as a JSON object) at the end of aggregation. This function uses the t-Digest algorithm.

Since: 0.2.0

def approx_percentile_combine ( state: Column ) : Column

Combines (merges) percentile input states into a single output state.

This allows scenarios where APPROX_PERCENTILE_ACCUMULATE is run over horizontal partitions of the same table, producing an algorithm state for each table partition. These states can later be combined using APPROX_PERCENTILE_COMBINE, producing the same output state as a single run of APPROX_PERCENTILE_ACCUMULATE over the entire table.

Since: 0.2.0

def approx_percentile_estimate ( state: Column , percentile: Double ) : Column

Returns the desired approximated percentile value for the specified t-Digest state.

Returns the desired approximated percentile value for the specified t-Digest state. APPROX_PERCENTILE_ESTIMATE(APPROX_PERCENTILE_ACCUMULATE(.)) is equivalent to APPROX_PERCENTILE(.).

Since: 0.2.0

def array_agg ( col: Column ) : Column

Returns the input values, pivoted into an ARRAY.

Returns the input values, pivoted into an ARRAY. If the input is empty, an empty ARRAY is returned.

Since: 0.2.0

def array_append ( array: Column , element: Column ) : Column

Returns an ARRAY containing all elements from the source ARRAYas well as the new element.

Returns an ARRAY containing all elements from the source ARRAYas well as the new element. The new element is located at end of the ARRAY.

array: The column containing the source ARRAY.
element: The column containing the element to be appended. The element may be of almost any data type. The data type does not need to match the data type(s) of the existing elements in the ARRAY.

Since: 0.2.0

def array_cat ( array1: Column , array2: Column ) : Column

Returns the concatenation of two ARRAYs.

array1: Column containing the source ARRAY.
array2: Column containing the ARRAY to be appended to array1 .

Since: 0.2.0

def array_compact ( array: Column ) : Column

Returns a compacted ARRAY with missing and null values removed, effectively converting sparse arrays into dense arrays.

Since: 0.2.0

def array_construct ( cols: Column * ) : Column

Returns an ARRAY constructed from zero, one, or more inputs.

cols: Columns containing the values (or expressions that evaluate to values). The values do not all need to be of the same data type.

Since: 0.2.0

def array_construct_compact ( cols: Column * ) : Column

Returns an ARRAY constructed from zero, one, or more inputs; the constructed ARRAY omits any NULL input values.

cols: Columns containing the values (or expressions that evaluate to values). The values do not all need to be of the same data type.

Since: 0.2.0

def array_contains ( variant: Column , array: Column ) : Column

Returns true if the specified VARIANT is found in the specified ARRAY.

variant: Column containing the VARIANT to find.
array: Column containing the ARRAY to search.

Since: 0.2.0

def array_insert ( array: Column , pos: Column , element: Column ) : Column

Returns an ARRAY containing all elements from the source ARRAY as well as the new element.

array: Column containing the source ARRAY.
pos: Column containing a (zero-based) position in the source ARRAY. The new element is inserted at this position. The original element from this position (if any) and all subsequent elements (if any) are shifted by one position to the right in the resulting array (i.e. inserting at position 0 has the same effect as using array_prepend ). A negative position is interpreted as an index from the back of the array (e.g. -1 results in insertion before the last element in the array).
element: Column containing the element to be inserted. The new element is located at position pos . The relative order of the other elements from the source array is preserved.

Since: 0.2.0

def array_intersection ( col1: Column , col2: Column ) : Column

Returns an ARRAY that contains the matching elements in the two input ARRAYs.

Since: 0.1.0

def array_position ( variant: Column , array: Column ) : Column

Returns the index of the first occurrence of an element in an ARRAY.

variant: Column containing the VARIANT value that you want to find. The function searches for the first occurrence of this value in the array.
array: Column containing the ARRAY to be searched.

Since: 0.2.0

def array_prepend ( array: Column , element: Column ) : Column

Returns an ARRAY containing the new element as well as all elements from the source ARRAY.

Returns an ARRAY containing the new element as well as all elements from the source ARRAY. The new element is positioned at the beginning of the ARRAY.

array: Column containing the source ARRAY.
element: Column containing the element to be prepended.

Since: 0.2.0

def array_size ( array: Column ) : Column

Returns the size of the input ARRAY.

If the specified column contains a VARIANT value that contains an ARRAY, the size of the ARRAY is returned; otherwise, NULL is returned if the value is not an ARRAY.

Since: 0.2.0

def array_slice ( array: Column , from: Column , to: Column ) : Column

Returns an ARRAY constructed from a specified subset of elements of the input ARRAY.

array: Column containing the source ARRAY.
from: Column containing a position in the source ARRAY. The position of the first element is 0 . Elements from positions less than this parameter are not included in the resulting ARRAY.
to: Column containing a position in the source ARRAY. Elements from positions equal to or greater than this parameter are not included in the resulting array.

Since: 0.2.0

def array_to_string ( array: Column , separator: Column ) : Column

Returns an input ARRAY converted to a string by casting all values to strings (using TO_VARCHAR) and concatenating them (using the string from the second argument to separate the elements).

array: Column containing the ARRAY of elements to convert to a string.
separator: Column containing the string to put between each element (e.g. a space, comma, or other human-readable separator).

Since: 0.2.0

def arrays_overlap ( a1: Column , a2: Column ) : Column

Compares whether two arrays have at least one element in common.

Compares whether two arrays have at least one element in common. Returns TRUE if there is at least one element in common; otherwise returns FALSE. The function is NULL-safe, meaning it treats NULLs as known values for comparing equality.

Since: 0.1.0

final def asInstanceOf [ T0 ] : T0

Definition Classes: Any

def as_array ( variant: Column ) : Column

Casts a VARIANT value to an array.

Since: 0.2.0

def as_binary ( variant: Column ) : Column

Casts a VARIANT value to a binary string.

Since: 0.2.0

def as_char ( variant: Column ) : Column

Casts a VARIANT value to a string.

Casts a VARIANT value to a string. Does not convert values of other types into string.

Since: 0.2.0

def as_date ( variant: Column ) : Column

Casts a VARIANT value to a date.

Casts a VARIANT value to a date. Does not convert from timestamps.

Since: 0.2.0

def as_decimal ( variant: Column , precision: Int , scale: Int ) : Column

Casts a VARIANT value to a fixed-point decimal (does not match floating-point values), with precision and scale.

Since: 0.2.0

def as_decimal ( variant: Column , precision: Int ) : Column

Casts a VARIANT value to a fixed-point decimal (does not match floating-point values), with precision.

Since: 0.2.0

def as_decimal ( variant: Column ) : Column

Casts a VARIANT value to a fixed-point decimal (does not match floating-point values).

Since: 0.2.0

def as_double ( variant: Column ) : Column

Casts a VARIANT value to a floating-point value.

Since: 0.2.0

def as_integer ( variant: Column ) : Column

Casts a VARIANT value to an integer.

Casts a VARIANT value to an integer. Does not match non-integer values.

Since: 0.2.0

def as_number ( variant: Column , precision: Int , scale: Int ) : Column

Casts a VARIANT value to a fixed-point decimal (does not match floating-point values), with precision and scale.

Since: 0.2.0

def as_number ( variant: Column , precision: Int ) : Column

Casts a VARIANT value to a fixed-point decimal (does not match floating-point values), with precision.

Since: 0.2.0

def as_number ( variant: Column ) : Column

Casts a VARIANT value to a fixed-point decimal (does not match floating-point values).

Since: 0.2.0

def as_object ( variant: Column ) : Column

Casts a VARIANT value to an object.

Since: 0.2.0

def as_real ( variant: Column ) : Column

Casts a VARIANT value to a floating-point value.

Since: 0.2.0

def as_time ( variant: Column ) : Column

Casts a VARIANT value to a time value.

Casts a VARIANT value to a time value. Does not convert from timestamps.

Since: 0.2.0

def as_timestamp_ltz ( variant: Column ) : Column

Casts a VARIANT value to a TIMESTAMP value with local timezone.

Since: 0.2.0

def as_timestamp_ntz ( variant: Column ) : Column

Casts a VARIANT value to a TIMESTAMP value with no timezone.

Since: 0.2.0

def as_timestamp_tz ( variant: Column ) : Column

Casts a VARIANT value to a TIMESTAMP value with timezone.

Since: 0.2.0

def as_varchar ( variant: Column ) : Column

Casts a VARIANT value to a string.

Casts a VARIANT value to a string. Does not convert values of other types into string.

Since: 0.2.0

def ascii ( e: Column ) : Column

Returns the ASCII code for the first character of a string.

Returns the ASCII code for the first character of a string. If the string is empty, a value of 0 is returned.

Since: 0.1.0

def asin ( e: Column ) : Column

Computes the inverse sine (arc sine) of its argument; the result is a number in the interval [-pi, pi].

Since: 0.1.0

def atan ( e: Column ) : Column

Computes the inverse tangent (arc tangent) of its argument; the result is a number in the interval [-pi, pi].

Since: 0.1.0

def atan2 ( y: Column , x: Column ) : Column

Computes the inverse tangent (arc tangent) of the ratio of its two arguments.

Since: 0.1.0

def avg ( e: Column ) : Column

Returns the average of non-NULL records.

Returns the average of non-NULL records. If all records inside a group are NULL, the function returns NULL.

Since: 0.1.0

def bitnot ( e: Column ) : Column

Returns the bitwise negation of a numeric expression.

Since: 0.1.0

def bitshiftleft ( e: Column , numBits: Column ) : Column

Shifts the bits for a numeric expression numBits positions to the left.

Since: 0.1.0

def bitshiftright ( e: Column , numBits: Column ) : Column

Shifts the bits for a numeric expression numBits positions to the right.

Since: 0.1.0

def callBuiltin ( functionName: String , args: Any * ) : Column

Invokes a built-in snowflake function with the specified name and arguments.

Invokes a built-in snowflake function with the specified name and arguments. Arguments can be of two types

a. Column , or

b. Basic types such as Int, Long, Double, Decimal etc. which are converted to Snowpark literals.

Since: 0.1.0

def callUDF ( udfName: String , cols: Any * ) : Column

Calls a user-defined function (UDF) by name.

Since: 0.1.0

def ceil ( e: Column ) : Column

Returns values from the specified column rounded to the nearest equal or larger integer.

Since: 0.1.0

def char ( col: Column ) : Column

Converts a Unicode code point (including 7-bit ASCII) into the character that matches the input Unicode.

Since: 0.1.0

def charindex ( targetExpr: Column , sourceExpr: Column , position: Column ) : Column

Searches for targetExpr in sourceExpr and, if successful, returns the position (1-based) of the targetExpr in sourceExpr.

Since: 0.1.0

def charindex ( targetExpr: Column , sourceExpr: Column ) : Column

Searches for targetExpr in sourceExpr and, if successful, returns the position (1-based) of the targetExpr in sourceExpr.

Since: 0.1.0

def check_json ( col: Column ) : Column

Checks the validity of a JSON document.

Checks the validity of a JSON document. If the input string is a valid JSON document or a NULL (i.e. no error would occur when parsing the input string), the function returns NULL. In case of a JSON parsing error, the function returns a string that contains the error message.

Since: 0.2.0

def check_xml ( col: Column ) : Column

Checks the validity of an XML document.

Checks the validity of an XML document. If the input string is a valid XML document or a NULL (i.e. no error would occur when parsing the input string), the function returns NULL. In case of an XML parsing error, the output string contains the error message.

Since: 0.2.0

def clone () : AnyRef

Attributes: protected[ lang ]
Definition Classes: AnyRef
Annotations: @throws ( ... ) @native () @HotSpotIntrinsicCandidate ()

def coalesce ( e: Column * ) : Column

Returns the first non-NULL expression among its arguments, or NULL if all its arguments are NULL.

Since: 0.1.0

def col ( df: DataFrame ) : Column

Generate a Column representing the result of the input DataFrame.

Generate a Column representing the result of the input DataFrame. The parameter df should have one column and must produce one row. Is an alias of toScalar .

For Example:

import functions._
val df1 = session.sql("select * from values(1,1,1),(2,2,3) as T(c1,c2,c3)")
val df2 = session.sql("select * from values(2) as T(a)")
df1.select(Column("c1"), col(df2)).show()
df1.filter(Column("c1") < col(df2)).show()

Since: 0.2.0

def col ( colName: String ) : Column

Returns the Column with the specified name.

Since: 0.1.0

def collate ( expr: Column , collationSpec: String ) : Column

Returns a copy of expr, but with the specified collationSpec property instead of the original collation specification property.

Collation Specification is specified here

Since: 0.1.0

def collation ( expr: Column ) : Column

Returns the collation specification of expr.

Since: 0.1.0

def column ( colName: String ) : Column

Returns a Column with the specified name.

Returns a Column with the specified name. Alias for col.

Since: 0.1.0

def concat ( exprs: Column * ) : Column

Concatenates one or more strings, or concatenates one or more binary values.

Concatenates one or more strings, or concatenates one or more binary values. If any of the values is null, the result is also null.

Since: 0.1.0

def concat_ws ( separator: Column , exprs: Column * ) : Column

Concatenates two or more strings, or concatenates two or more binary values.

Concatenates two or more strings, or concatenates two or more binary values. If any of the values is null, the result is also null.

Since: 0.1.0

def contains ( col: Column , str: Column ) : Column

Returns true if col contains str.

Since: 0.1.0

def convert_timezone ( targetTimeZone: Column , sourceTimestamp: Column ) : Column

Converts the given sourceTimestampNTZ to targetTimeZone.

Supported time zones are listed here

Example

timestamp.select(convert_timezone(lit("America/New_York"), col("time")))

Since: 0.1.0

def convert_timezone ( sourceTimeZone: Column , targetTimeZone: Column , sourceTimestampNTZ: Column ) : Column

Converts the given sourceTimestampNTZ from sourceTimeZone to targetTimeZone.

Supported time zones are listed here

Example

timestampNTZ.select(convert_timezone(lit("America/Los_Angeles"), lit("America/New_York"), col("time")))

Since: 0.1.0

def corr ( column1: Column , column2: Column ) : Column

Returns the correlation coefficient for non-null pairs in a group.

Since: 0.1.0

def cos ( e: Column ) : Column

Computes the cosine of its argument; the argument should be expressed in radians.

Since: 0.1.0

def cosh ( e: Column ) : Column

Computes the hyperbolic cosine of its argument.

Since: 0.1.0

def count ( e: Column ) : Column

Returns either the number of non-NULL records for the specified columns, or the total number of records.

Since: 0.1.0

def countDistinct ( expr: Column , exprs: Column * ) : Column

Returns either the number of non-NULL distinct records for the specified columns, or the total number of the distinct records.

Returns either the number of non-NULL distinct records for the specified columns, or the total number of the distinct records. An alias of count_distinct.

Since: 1.13.0

def countDistinct ( colName: String , colNames: String * ) : Column

Returns either the number of non-NULL distinct records for the specified columns, or the total number of the distinct records.

Returns either the number of non-NULL distinct records for the specified columns, or the total number of the distinct records. An alias of count_distinct.

Since: 1.13.0

def count_distinct ( expr: Column , exprs: Column * ) : Column

Returns either the number of non-NULL distinct records for the specified columns, or the total number of the distinct records.

Since: 0.1.0

def covar_pop ( column1: Column , column2: Column ) : Column

Returns the population covariance for non-null pairs in a group.

Since: 0.1.0

def covar_samp ( column1: Column , column2: Column ) : Column

Returns the sample covariance for non-null pairs in a group.

Since: 0.1.0

def cume_dist () : Column

Finds the cumulative distribution of a value with regard to other values within the same window partition.

Since: 0.1.0

def current_account () : Column

Returns the account used by the user's current session.

Since: 0.1.0

def current_available_roles () : Column

Returns a JSON string that lists all roles granted to the current user.

Since: 0.1.0

def current_database () : Column

Returns the name of the database in use for the current session.

Since: 0.1.0

def current_date () : Column

Returns the current date of the system.

Since: 0.1.0

def current_region () : Column

Returns the name of the region for the account where the current user is logged in.

Since: 0.1.0

def current_role () : Column

Returns the name of the role in use for the current session.

Since: 0.1.0

def current_schema () : Column

Returns the name of the schema in use by the current session.

Since: 0.1.0

def current_schemas () : Column

Returns active search path schemas.

Since: 0.1.0

def current_session () : Column

Returns a unique system identifier for the Snowflake session corresponding to the present connection.

Since: 0.1.0

def current_statement () : Column

Returns the SQL text of the statement that is currently executing.

Since: 0.1.0

def current_time () : Column

Returns the current time for the system.

Since: 0.1.0

def current_timestamp () : Column

Returns the current timestamp for the system.

Since: 0.1.0

def current_user () : Column

Returns the name of the user currently logged into the system.

Since: 0.1.0

def current_version () : Column

Returns the current Snowflake version.

Since: 0.1.0

def current_warehouse () : Column

Returns the name of the warehouse in use for the current session.

Since: 0.1.0

def date_from_parts ( year: Column , month: Column , day: Column ) : Column

Creates a date from individual numeric components that represent the year, month, and day of the month.

Since: 0.1.0

def date_trunc ( format: String , timestamp: Column ) : Column

Truncates a DATE, TIME, or TIMESTAMP to the specified precision.

Since: 0.1.0

def dateadd ( part: String , value: Column , expr: Column ) : Column

Adds the specified value for the specified date or time art to date or time expr.

Supported date and time parts are listed here

Example: add one year on dates

date.select(dateadd("year", lit(1), col("date_col")))

Since: 0.1.0

def datediff ( part: String , col1: Column , col2: Column ) : Column

Calculates the difference between two date, time, or timestamp columns based on the date or time part requested.

Supported date and time parts are listed here

Example: year difference between two date columns

date.select(datediff("year", col("date_col1"), col("date_col2"))),

Since: 0.1.0

def dayname ( expr: Column ) : Column

Extracts the three-letter day-of-week name from the specified date or timestamp.

Since: 0.1.0

def dayofmonth ( e: Column ) : Column

Extracts the day of month from a date or timestamp.

Since: 0.1.0

def dayofweek ( e: Column ) : Column

Extracts the day of week from a date or timestamp.

Since: 0.1.0

def dayofyear ( e: Column ) : Column

Extracts the day of year from a date or timestamp.

Since: 0.1.0

def degrees ( e: Column ) : Column

Converts radians to degrees.

Since: 0.1.0

def dense_rank () : Column

Returns the rank of a value within a group of values, without gaps in the ranks.

Returns the rank of a value within a group of values, without gaps in the ranks. The rank value starts at 1 and continues up sequentially. If two values are the same, they will have the same rank.

Since: 0.1.0

def div0 ( dividend: Column , divisor: Column ) : Column

Performs division like the division operator (/), but returns 0 when the divisor is 0 (rather than reporting an error).

Since: 0.1.0

def endswith ( expr: Column , str: Column ) : Column

Returns TRUE if expr ends with str.

Since: 0.1.0

final def eq ( arg0: AnyRef ) : Boolean

Definition Classes: AnyRef

def equal_nan ( e: Column ) : Column

Return true if the value in the column is not a number (NaN).

Since: 0.1.0

def equals ( arg0: Any ) : Boolean

Definition Classes: AnyRef → Any

def exp ( e: Column ) : Column

Computes Euler's number e raised to a floating-point value.

Since: 0.1.0

def factorial ( e: Column ) : Column

Computes the factorial of its input.

Computes the factorial of its input. The input argument must be an integer expression in the range of 0 to 33.

Since: 0.1.0

def floor ( e: Column ) : Column

Returns values from the specified column rounded to the nearest equal or smaller integer.

Since: 0.1.0

def get ( col1: Column , col2: Column ) : Column

Extracts a value from an object or array; returns NULL if either of the arguments is NULL.

Since: 0.2.0

final def getClass () : Class [_]

Definition Classes: AnyRef → Any
Annotations: @native () @HotSpotIntrinsicCandidate ()

def get_ignore_case ( obj: Column , field: Column ) : Column

Extracts a field value from an object; returns NULL if either of the arguments is NULL.

Extracts a field value from an object; returns NULL if either of the arguments is NULL. This function is similar to GET but applies case-insensitive matching to field names.

Since: 0.2.0

def get_path ( col: Column , path: Column ) : Column

Extracts a value from semi-structured data using a path name.

Since: 0.2.0

def greatest ( exprs: Column * ) : Column

Returns the largest value from a list of expressions.

Returns the largest value from a list of expressions. If any of the argument values is NULL, the result is NULL. GREATEST supports all data types, including VARIANT.

Since: 0.1.0

def grouping ( e: Column ) : Column

Describes which of a list of expressions are grouped in a row produced by a GROUP BY query.

Since: 0.1.0

def grouping_id ( cols: Column * ) : Column

Describes which of a list of expressions are grouped in a row produced by a GROUP BY query.

Since: 0.1.0

def hash ( cols: Column * ) : Column

Returns a signed 64-bit hash value.

Returns a signed 64-bit hash value. Note that HASH never returns NULL, even for NULL inputs.

Since: 0.1.0

def hashCode () : Int

Definition Classes: AnyRef → Any
Annotations: @native () @HotSpotIntrinsicCandidate ()

def hour ( e: Column ) : Column

Extracts the hour from a date or timestamp.

Since: 0.1.0

def iff ( condition: Column , expr1: Column , expr2: Column ) : Column

Returns one of two specified expressions, depending on a condition.

This is equivalent to an if-then-else expression. If condition evaluates to TRUE, the function returns expr1 . Otherwise, the function returns expr2 .

condition: The condition to evaluate.
expr1: The expression to return if the condition evaluates to TRUE.
expr2: The expression to return if the condition is not TRUE (i.e. if it is FALSE or NULL).

Since: 0.9.0

def in ( columns: Seq [ Column ] , df: DataFrame ) : Column

Returns a conditional expression that you can pass to the filter or where method to perform the equivalent of a WHERE ...

Returns a conditional expression that you can pass to the filter or where method to perform the equivalent of a WHERE ... IN query with the subquery represented by the specified DataFrame.

The expression evaluates to true if the value in the column is one of the values in the column of the same name in a specified DataFrame.

For example, the following code returns a DataFrame that contains the rows where the values of the columns c1 and c2 in df2 match the values of the columns a and b in df1 . This is equivalent to SELECT * FROM table2 WHERE (c1, c2) IN (SELECT a, b FROM table1).

val df1 = session.sql("select a, b from table1").
val df2 = session.table(table2)
val dfFilter = df2.filter(functions.in(Seq(col("c1"), col("c2")), df1))

columns: A sequence of the columns to compare for the IN operation.
df: The DataFrame used as the values for the IN operation

Since: 0.10.0

def in ( columns: Seq [ Column ] , values: Seq [ Seq [ Any ]] ) : Column

Returns a conditional expression that you can pass to the filter or where method to perform the equivalent of a WHERE ...

Returns a conditional expression that you can pass to the filter or where method to perform the equivalent of a WHERE ... IN query that matches rows containing a sequence of values.

The expression evaluates to true if the values in a row matches the values in one of the specified sequences.

For example, the following code returns a DataFrame that contains the rows in which the columns c1 and c2 contain the values: - 1 and "a" , or - 2 and "b" This is equivalent to SELECT * FROM table WHERE (c1, c2) IN ((1, 'a'), (2, 'b')) .

val df2 = df.filter(functions.in(Seq(df("c1"), df("c2")), Seq(Seq(1, "a"), Seq(2, "b"))))

columns: A sequence of the columns to compare for the IN operation.
values: A sequence containing the sequences of values to compare for the IN operation.

Since: 0.10.0

def initcap ( e: Column ) : Column

Returns the input string with the first letter of each word in uppercase and the subsequent letters in lowercase.

Since: 0.1.0

def insert ( baseExpr: Column , position: Column , length: Column , insertExpr: Column ) : Column

Replaces a substring of the specified length, starting at the specified position, with a new string or binary value.

Since: 0.1.0

final def isInstanceOf [ T0 ] : Boolean

Definition Classes: Any

def is_array ( col: Column ) : Column

Returns true if the specified VARIANT column contains an ARRAY value.

Since: 0.1.0

def is_binary ( col: Column ) : Column

Returns true if the specified VARIANT column contains a binary value.

Since: 0.1.0

def is_boolean ( col: Column ) : Column

Returns true if the specified VARIANT column contains a Boolean value.

Since: 0.1.0

def is_char ( col: Column ) : Column

Returns true if the specified VARIANT column contains a string value.

Since: 0.1.0

def is_date ( col: Column ) : Column

Returns true if the specified VARIANT column contains a DATE value.

Since: 0.1.0

def is_date_value ( col: Column ) : Column

Returns true if the specified VARIANT column contains a DATE value.

Since: 0.1.0

def is_decimal ( col: Column ) : Column

Returns true if the specified VARIANT column contains a fixed-point decimal value or integer.

Since: 0.1.0

def is_double ( col: Column ) : Column

Returns true if the specified VARIANT column contains a floating-point value, fixed-point decimal, or integer.

Since: 0.1.0

def is_integer ( col: Column ) : Column

Returns true if the specified VARIANT column contains an integer value.

Since: 0.1.0

def is_null ( e: Column ) : Column

Return true if the value in the column is null.

Since: 0.1.0

def is_null_value ( col: Column ) : Column

Returns true if the specified VARIANT column is a JSON null value.

Since: 0.1.0

def is_object ( col: Column ) : Column

Returns true if the specified VARIANT column contains an OBJECT value.

Since: 0.1.0

def is_real ( col: Column ) : Column

Returns true if the specified VARIANT column contains a floating-point value, fixed-point decimal, or integer.

Since: 0.1.0

def is_time ( col: Column ) : Column

Returns true if the specified VARIANT column contains a TIME value.

Since: 0.1.0

def is_timestamp_ltz ( col: Column ) : Column

Returns true if the specified VARIANT column contains a TIMESTAMP value to be interpreted using the local time zone.

Since: 0.1.0

def is_timestamp_ntz ( col: Column ) : Column

Returns true if the specified VARIANT column contains a TIMESTAMP value with no time zone.

Since: 0.1.0

def is_timestamp_tz ( col: Column ) : Column

Returns true if the specified VARIANT column contains a TIMESTAMP value with a time zone.

Since: 0.1.0

def is_varchar ( col: Column ) : Column

Returns true if the specified VARIANT column contains a string value.

Since: 0.1.0

def json_extract_path_text ( col: Column , path: Column ) : Column

Parses a JSON string and returns the value of an element at a specified path in the resulting JSON document.

col: Column containing the JSON string that should be parsed.
path: Column containing the path to the element that should be extracted.

Since: 0.2.0

def kurtosis ( e: Column ) : Column

Returns the population excess kurtosis of non-NULL records.

Returns the population excess kurtosis of non-NULL records. If all records inside a group are NULL, the function returns NULL.

Since: 0.1.0

def lag ( e: Column ) : Column

Accesses data in a previous row in the same result set without having to join the table to itself.

Since: 0.1.0

def lag ( e: Column , offset: Int ) : Column

Accesses data in a previous row in the same result set without having to join the table to itself.

Since: 0.1.0

def lag ( e: Column , offset: Int , defaultValue: Column ) : Column

Accesses data in a previous row in the same result set without having to join the table to itself.

Since: 0.1.0

def last_day ( e: Column ) : Column

Returns the last day of the specified date part for a date or timestamp.

Returns the last day of the specified date part for a date or timestamp. Commonly used to return the last day of the month for a date or timestamp.

Since: 0.1.0

def lead ( e: Column ) : Column

Accesses data in a subsequent row in the same result set without having to join the table to itself.

Since: 0.1.0

def lead ( e: Column , offset: Int ) : Column

Accesses data in a subsequent row in the same result set without having to join the table to itself.

Since: 0.1.0

def lead ( e: Column , offset: Int , defaultValue: Column ) : Column

Accesses data in a subsequent row in the same result set without having to join the table to itself.

Since: 0.1.0

def least ( exprs: Column * ) : Column

Returns the smallest value from a list of expressions.

Returns the smallest value from a list of expressions. LEAST supports all data types, including VARIANT.

Since: 0.1.0

def left ( strExpr: Column , lengthExpr: Column ) : Column

Returns a left most substring of strExpr.

Since: 0.1.0

def length ( e: Column ) : Column

Returns the length of an input string or binary value.

Returns the length of an input string or binary value. For strings, the length is the number of characters, and UTF-8 characters are counted as a single character. For binary, the length is the number of bytes.

Since: 0.1.0

def listagg ( col: Column ) : Column

Returns the concatenated input values, separated by empty string.

For example:

df.groupBy(df.col("col1")).agg(listagg(df.col("col2"), ",")
    .withinGroup(df.col("col2").asc))

df.select(listagg(df.col("col2"), ",", false))

col: The expression (typically a Column) that determines the values to be put into the list. The expression should evaluate to a string, or to a data type that can be cast to string.

Since: 0.12.0

def listagg ( col: Column , delimiter: String ) : Column

Returns the concatenated input values, separated by delimiter string.

For example:

df.groupBy(df.col("col1")).agg(listagg(df.col("col2"), ",")
    .withinGroup(df.col("col2").asc))

df.select(listagg(df.col("col2"), ",", false))

col: The expression (typically a Column) that determines the values to be put into the list. The expression should evaluate to a string, or to a data type that can be cast to string.
delimiter: A string delimiter.

Since: 0.12.0

def listagg ( col: Column , delimiter: String , isDistinct: Boolean ) : Column

Returns the concatenated input values, separated by delimiter string.

For example:

df.groupBy(df.col("col1")).agg(listagg(df.col("col2"), ",")
    .withinGroup(df.col("col2").asc))

df.select(listagg(df.col("col2"), ",", false))

col: The expression (typically a Column) that determines the values to be put into the list. The expression should evaluate to a string, or to a data type that can be cast to string.
delimiter: A string delimiter.
isDistinct: Whether the input expression is distinct.

Since: 0.12.0

def lit ( literal: Any ) : Column

Creates a Column expression for a literal value.

Since: 0.1.0

def log ( base: Column , a: Column ) : Column

Returns the logarithm of a numeric expression.

Since: 0.1.0

def lower ( e: Column ) : Column

Returns the input string with all characters converted to lowercase.

Since: 0.1.0

def lpad ( str: Column , len: Column , pad: Column ) : Column

Left-pads a string with characters from another string, or left-pads a binary value with bytes from another binary value.

Since: 0.1.0

def ltrim ( e: Column ) : Column

Removes leading characters, including whitespace, from a string.

Since: 0.1.0

def ltrim ( e: Column , trimString: Column ) : Column

Removes leading characters, including whitespace, from a string.

Since: 0.1.0

def max ( e: Column ) : Column

Returns the maximum value for the records in a group.

Returns the maximum value for the records in a group. NULL values are ignored unless all the records are NULL, in which case a NULL value is returned.

Since: 0.1.0

def max ( colName: String ) : Column

Returns the maximum value for the records in a group.

Returns the maximum value for the records in a group. NULL values are ignored unless all the records are NULL, in which case a NULL value is returned.

Example:

val df = session.createDataFrame(Seq(1, 3, 10, 1, 3)).toDF("x")
df.select(max("x")).show()

----------------
|"MAX(""X"")"  |
----------------
|10            |
----------------

colName: The name of the column
returns: The maximum value of the given column

Since: 1.13.0

def md5 ( e: Column ) : Column

Returns a 32-character hex-encoded string containing the 128-bit MD5 message digest.

Since: 0.1.0

def mean ( e: Column ) : Column

Returns the average of non-NULL records.

Returns the average of non-NULL records. If all records inside a group are NULL, the function returns NULL. Alias of avg

Since: 0.1.0

def mean ( colName: String ) : Column

Returns the average of non-NULL records.

Returns the average of non-NULL records. If all records inside a group are NULL, the function returns NULL. Alias of avg.

Example:

val df = session.createDataFrame(Seq(1, 3, 10, 1, 3)).toDF("x")
df.select(mean("x")).show()

----------------
|"AVG(""X"")"  |
----------------
|3.600000      |
----------------

colName: The name of the column
returns: The average value of the given column

Since: 1.13.0

def median ( e: Column ) : Column

Returns the median value for the records in a group.

Returns the median value for the records in a group. NULL values are ignored unless all the records are NULL, in which case a NULL value is returned.

Since: 0.5.0

def min ( e: Column ) : Column

Returns the minimum value for the records in a group.

Returns the minimum value for the records in a group. NULL values are ignored unless all the records are NULL, in which case a NULL value is returned.

Since: 0.1.0

def min ( colName: String ) : Column

Returns the minimum value for the records in a group.

Returns the minimum value for the records in a group. NULL values are ignored unless all the records are NULL, in which case a NULL value is returned.

Example:

val df = session.createDataFrame(Seq(1, 3, 10, 1, 3)).toDF("x")
df.select(min("x")).show()

----------------
|"MIN(""X"")"  |
----------------
|1             |
----------------

colName: The name of the column
returns: The minimum value of the given column

Since: 1.13.0

def minute ( e: Column ) : Column

Extracts the minute from a date or timestamp.

Since: 0.1.0

def month ( e: Column ) : Column

Extracts the month from a date or timestamp.

Since: 0.1.0

def monthname ( expr: Column ) : Column

Extracts the three-letter month name from the specified date or timestamp.

Since: 0.1.0

final def ne ( arg0: AnyRef ) : Boolean

Definition Classes: AnyRef

def negate ( e: Column ) : Column

Returns the negation of the value in the column (equivalent to a unary minus).

Since: 0.1.0

def next_day ( date: Column , dayOfWeek: Column ) : Column

Returns the date of the first specified DOW (day of week) that occurs after the input date.

Since: 0.1.0

def not ( e: Column ) : Column

Returns the inverse of a boolean expression.

Since: 0.1.0

final def notify () : Unit

Definition Classes: AnyRef
Annotations: @native () @HotSpotIntrinsicCandidate ()

final def notifyAll () : Unit

Definition Classes: AnyRef
Annotations: @native () @HotSpotIntrinsicCandidate ()

def ntile ( n: Column ) : Column

Divides an ordered data set equally into the number of buckets specified by n.

Divides an ordered data set equally into the number of buckets specified by n. Buckets are sequentially numbered 1 through n.

Since: 0.1.0

def object_construct ( key_values: Column * ) : Column

Returns an OBJECT constructed from the arguments.

Since: 0.2.0

def object_delete ( obj: Column , key1: Column , keys: Column * ) : Column

Returns an object containing the contents of the input (i.e.source) object with one or more keys removed.

Since: 0.2.0

def object_insert ( obj: Column , key: Column , value: Column , update_flag: Column ) : Column

Returns an object consisting of the input object with a new key-value pair inserted (or an existing key updated with a new value).

Since: 0.2.0

def object_insert ( obj: Column , key: Column , value: Column ) : Column

Returns an object consisting of the input object with a new key-value pair inserted.

Returns an object consisting of the input object with a new key-value pair inserted. The input key must not exist in the object.

Since: 0.2.0

def object_keys ( obj: Column ) : Column

Returns an array containing the list of keys in the input object.

Since: 0.2.0

def object_pick ( obj: Column , key1: Column , keys: Column * ) : Column

Returns a new OBJECT containing some of the key-value pairs from an existing object.

To identify the key-value pairs to include in the new object, pass in the keys as arguments, or pass in an array containing the keys.

If a specified key is not present in the input object, the key is ignored.

Since: 0.2.0

def objectagg ( key: Column , value: Column ) : Column

Returns one OBJECT per group.

Returns one OBJECT per group. For each (key, value) input pair, where key must be a VARCHAR and value must be a VARIANT, the resulting OBJECT contains a key:value field.

Since: 0.2.0

def parse_json ( col: Column ) : Column

Parse the value of the specified column as a JSON string and returns the resulting JSON document.

Since: 0.2.0

def parse_xml ( col: Column ) : Column

Parse the value of the specified column as a JSON string and returns the resulting XML document.

Since: 0.2.0

def percent_rank () : Column

Returns the relative rank of a value within a group of values, specified as a percentage ranging from 0.0 to 1.0.

Since: 0.1.0

def pow ( l: Column , r: Column ) : Column

Returns a number (l) raised to the specified power (r).

Since: 0.1.0

def previous_day ( date: Column , dayOfWeek: Column ) : Column

Returns the date of the first specified DOW (day of week) that occurs before the input date.

Since: 0.1.0

def quarter ( e: Column ) : Column

Extracts the quarter from a date or timestamp.

Since: 0.1.0

def radians ( e: Column ) : Column

Converts degrees to radians.

Since: 0.1.0

def random () : Column

Each call returns a pseudo-random 64-bit integer.

Since: 0.1.0

def random ( seed: Long ) : Column

Each call returns a pseudo-random 64-bit integer.

Since: 0.1.0

def rank () : Column

Returns the rank of a value within an ordered group of values.

Returns the rank of a value within an ordered group of values. The rank value starts at 1 and continues up.

Since: 0.1.0

def regexp_count ( strExpr: Column , pattern: Column ) : Column

Returns the number of times that a pattern occurs in a strExpr.

Pattern syntax is specified here

Parameter detail is specified here

Since: 0.1.0

def regexp_count ( strExpr: Column , pattern: Column , position: Column , parameters: Column ) : Column

Returns the number of times that a pattern occurs in a strExpr.

Pattern syntax is specified here

Parameter detail is specified here

Since: 0.1.0

def regexp_replace ( strExpr: Column , pattern: Column , replacement: Column ) : Column

Returns the subject with the specified pattern (or all occurrences of the pattern) replaced by a replacement string.

Returns the subject with the specified pattern (or all occurrences of the pattern) replaced by a replacement string. If no matches are found, returns the original subject.

Since: 1.9.0

def regexp_replace ( strExpr: Column , pattern: Column ) : Column

Returns the subject with the specified pattern (or all occurrences of the pattern) removed.

Returns the subject with the specified pattern (or all occurrences of the pattern) removed. If no matches are found, returns the original subject.

Since: 1.9.0

def repeat ( str: Column , n: Column ) : Column

Builds a string by repeating the input for the specified number of times.

Since: 0.1.0

def replace ( strExpr: Column , pattern: Column ) : Column

Removes all occurrences of a specified strExpr, and optionally replaces them with replacement.

Since: 0.1.0

def replace ( strExpr: Column , pattern: Column , replacement: Column ) : Column

Removes all occurrences of a specified strExpr, and optionally replaces them with replacement.

Since: 0.1.0

def right ( strExpr: Column , lengthExpr: Column ) : Column

Returns a right most substring of strExpr.

Since: 0.1.0

def round ( e: Column ) : Column

Returns rounded values for the specified column.

Since: 0.1.0

def round ( e: Column , scale: Column ) : Column

Returns rounded values for the specified column.

Since: 0.1.0

def row_number () : Column

Returns a unique row number for each row within a window partition.

Returns a unique row number for each row within a window partition. The row number starts at 1 and continues up sequentially.

Since: 0.1.0

def rpad ( str: Column , len: Column , pad: Column ) : Column

Right-pads a string with characters from another string, or right-pads a binary value with bytes from another binary value.

Since: 0.1.0

def rtrim ( e: Column ) : Column

Removes trailing characters, including whitespace, from a string.

Since: 0.1.0

def rtrim ( e: Column , trimString: Column ) : Column

Removes trailing characters, including whitespace, from a string.

Since: 0.1.0

def second ( e: Column ) : Column

Extracts the second from a date or timestamp.

Since: 0.1.0

def seq1 ( startsFromZero: Boolean ) : Column

Generates a sequence of monotonically increasing integers, with wrap-around.

Generates a sequence of monotonically increasing integers, with wrap-around. Wrap-around occurs after the largest representable integer of the integer width 1 byte.

startsFromZero: if true, the sequence continues at 0 after wrap-around, otherwise, continues at the smallest representable number based on the given integer width.

Since: 0.11.0

def seq1 () : Column

Generates a sequence of monotonically increasing integers, with wrap-around.

Generates a sequence of monotonically increasing integers, with wrap-around. Wrap-around occurs after the largest representable integer of the integer width 1 byte. the sequence continues at 0 after wrap-around.

Since: 0.11.0

def seq2 ( startsFromZero: Boolean ) : Column

Generates a sequence of monotonically increasing integers, with wrap-around.

Generates a sequence of monotonically increasing integers, with wrap-around. Wrap-around occurs after the largest representable integer of the integer width 2 byte.

startsFromZero: if true, the sequence continues at 0 after wrap-around, otherwise, continues at the smallest representable number based on the given integer width.

Since: 0.11.0

def seq2 () : Column

Generates a sequence of monotonically increasing integers, with wrap-around.

Generates a sequence of monotonically increasing integers, with wrap-around. Wrap-around occurs after the largest representable integer of the integer width 2 byte. the sequence continues at 0 after wrap-around.

Since: 0.11.0

def seq4 ( startsFromZero: Boolean ) : Column

Generates a sequence of monotonically increasing integers, with wrap-around.

Generates a sequence of monotonically increasing integers, with wrap-around. Wrap-around occurs after the largest representable integer of the integer width 4 byte.

startsFromZero: if true, the sequence continues at 0 after wrap-around, otherwise, continues at the smallest representable number based on the given integer width.

Since: 0.11.0

def seq4 () : Column

Generates a sequence of monotonically increasing integers, with wrap-around.

Generates a sequence of monotonically increasing integers, with wrap-around. Wrap-around occurs after the largest representable integer of the integer width 4 byte. the sequence continues at 0 after wrap-around.

Since: 0.11.0

def seq8 ( startsFromZero: Boolean ) : Column

Generates a sequence of monotonically increasing integers, with wrap-around.

Generates a sequence of monotonically increasing integers, with wrap-around. Wrap-around occurs after the largest representable integer of the integer width 8 byte.

startsFromZero: if true, the sequence continues at 0 after wrap-around, otherwise, continues at the smallest representable number based on the given integer width.

Since: 0.11.0

def seq8 () : Column

Generates a sequence of monotonically increasing integers, with wrap-around.

Generates a sequence of monotonically increasing integers, with wrap-around. Wrap-around occurs after the largest representable integer of the integer width 8 byte. the sequence continues at 0 after wrap-around.

Since: 0.11.0

def sha1 ( e: Column ) : Column

Returns a 40-character hex-encoded string containing the 160-bit SHA-1 message digest.

Since: 0.1.0

def sha2 ( e: Column , numBits: Int ) : Column

Returns a hex-encoded string containing the N-bit SHA-2 message digest, where N is the specified output digest size.

Since: 0.1.0

def sin ( e: Column ) : Column

Computes the sine of its argument; the argument should be expressed in radians.

Since: 0.1.0

def sinh ( e: Column ) : Column

Computes the hyperbolic sine of its argument.

Since: 0.1.0

def skew ( e: Column ) : Column

Returns the sample skewness of non-NULL records.

Returns the sample skewness of non-NULL records. If all records inside a group are NULL, the function returns NULL.

Since: 0.1.0

def soundex ( e: Column ) : Column

Returns a string that contains a phonetic representation of the input string.

Since: 0.1.0

def split ( str: Column , pattern: Column ) : Column

Splits a given string with a given separator and returns the result in an array of strings.

Splits a given string with a given separator and returns the result in an array of strings. To specify a string separator, use the lit() function.

Example 1:

val df = session.createDataFrame(
               Seq(("many-many-words", "-"), ("hello--hello", "--"))).toDF("V", "D")
df.select(split(col("V"), col("D"))).show()

|"SPLIT(""V"", ""D"")" | ------------------------- |[ | | "many", | | "many", | | "words" | |] | |[ | | "hello", | | "hello" | |] | -------------------------

Example 2:

val df = session.createDataFrame(Seq("many-many-words", "hello-hi-hello")).toDF("V")
df.select(split(col("V"), lit("-"))).show()

|"SPLIT(""V"", ""D"")" | ------------------------- |[ | | "many", | | "many", | | "words" | |] | |[ | | "hello", | | "hello" | |] | -------------------------

Since: 0.1.0

def sqlExpr ( sqlText: String ) : Column

Creates a Column expression from raw SQL text.

Note that the function does not interpret or check the SQL text.

Since: 0.1.0

def sqrt ( e: Column ) : Column

Returns the square-root of a non-negative numeric expression.

Since: 0.1.0

def startswith ( col: Column , str: Column ) : Column

Returns true if col starts with str.

Since: 0.1.0

def stddev ( e: Column ) : Column

Returns the sample standard deviation (square root of sample variance) of non-NULL values.

Returns the sample standard deviation (square root of sample variance) of non-NULL values. If all records inside a group are NULL, returns NULL.

Since: 0.1.0

def stddev_pop ( e: Column ) : Column

Returns the population standard deviation (square root of variance) of non-NULL values.

Returns the population standard deviation (square root of variance) of non-NULL values. If all records inside a group are NULL, returns NULL.

Since: 0.1.0

def stddev_samp ( e: Column ) : Column

Returns the sample standard deviation (square root of sample variance) of non-NULL values.

Returns the sample standard deviation (square root of sample variance) of non-NULL values. If all records inside a group are NULL, returns NULL. Alias of stddev

Since: 0.1.0

def strip_null_value ( col: Column ) : Column

Converts a JSON "null" value in the specified column to a SQL NULL value.

Converts a JSON "null" value in the specified column to a SQL NULL value. All other VARIANT values in the column are returned unchanged.

Since: 0.2.0

def strtok_to_array ( array: Column , delimiter: Column ) : Column

Tokenizes the given string using the given set of delimiters and returns the tokens as an array.

Tokenizes the given string using the given set of delimiters and returns the tokens as an array. If either parameter is a NULL, a NULL is returned. An empty array is returned if tokenization produces no tokens.

Since: 0.2.0

def strtok_to_array ( array: Column ) : Column

Tokenizes the given string using the given set of delimiters and returns the tokens as an array.

Tokenizes the given string using the given set of delimiters and returns the tokens as an array. If either parameter is a NULL, a NULL is returned. An empty array is returned if tokenization produces no tokens.

Since: 0.2.0

def substring ( str: Column , pos: Column , len: Column ) : Column

Returns the portion of the string or binary value str, starting from the character/byte specified by pos, with limited length.

Since: 0.1.0

def sum ( colName: String ) : Column

Returns the sum of non-NULL records in a group.

Returns the sum of non-NULL records in a group. If all records inside a group are NULL, the function returns NULL.

colName: The input column name
returns: The result column

Since: 1.12.0

def sum ( e: Column ) : Column

Returns the sum of non-NULL records in a group.

Returns the sum of non-NULL records in a group. If all records inside a group are NULL, the function returns NULL.

Since: 0.1.0

def sum_distinct ( e: Column ) : Column

Returns the sum of non-NULL distinct records in a group.

Returns the sum of non-NULL distinct records in a group. You can use the DISTINCT keyword to compute the sum of unique non-null values. If all records inside a group are NULL, the function returns NULL.

Since: 0.1.0

final def synchronized [ T0 ] ( arg0: ⇒ T0 ) : T0

Definition Classes: AnyRef

def sysdate () : Column

Returns the current timestamp for the system, but in the UTC time zone.

Since: 0.1.0

def tan ( e: Column ) : Column

Computes the tangent of its argument; the argument should be expressed in radians.

Since: 0.1.0

def tanh ( e: Column ) : Column

Computes the hyperbolic tangent of its argument.

Since: 0.1.0

def time_from_parts ( hour: Column , minute: Column , second: Column ) : Column

Creates a time from individual numeric components.

Since: 0.1.0

def time_from_parts ( hour: Column , minute: Column , second: Column , nanoseconds: Column ) : Column

Creates a time from individual numeric components.

Since: 0.1.0

def timestamp_from_parts ( dateExpr: Column , timeExpr: Column ) : Column