Analyzing Queries and Troubleshooting with Snowpark

This topic provides some guidelines on analyzing queries and troubleshooting problems when working with the Snowpark library.

In this Topic:

Viewing the Execution Plan for a Query in Snowpark

To inspect the evaluation plan of a DataFrame, call the explain method of the DataFrame. This prints the SQL statements used to evaluate the DataFrame. If there is only one SQL statement, the method also prints the logical plan for the statement.

----------DATAFRAME EXECUTION PLAN----------
Query List:
0.
SELECT
  "_1" AS "col %",
  "_2" AS "col *"
FROM
  (
    SELECT
      *
    FROM
      (
        VALUES
          (1 :: int, 2 :: int),
          (3 :: int, 4 :: int) AS SN_TEMP_OBJECT_639016133("_1", "_2")
      )
  )
Logical Execution Plan:
 GlobalStats:
    partitionsTotal=0
    partitionsAssigned=0
    bytesAssigned=0
Operations:
1:0     ->Result  SN_TEMP_OBJECT_639016133.COLUMN1, SN_TEMP_OBJECT_639016133.COLUMN2
1:1          ->ValuesClause  (1, 2), (3, 4)

--------------------------------------------

After the execution of a DataFrame has been triggered, you can check on the progress of the query in the History History tab page in the Snowflake web interface.

In the Query Tag column, you can find the name of the function and the line number in your code that triggered this query.

Snowpark request in the History page in the Snowflake web interface

Troubleshooting

Changing the Logging Settings

By default, the Snowpark library logs INFO level messages to stdout. If you want to change these logging settings, you can do this programmatically or in a properties file.

To change the logging level programmatically, call org.apache.log4j.Logger.getLogger("com.snowflake.snowpark") to get the Logger object for the library, and then call Logger.setLevel to change the level to one of the supported levels. For example:

import org.apache.log4j.{Level, Logger}
...
Logger.getLogger("com.snowflake.snowpark").setLevel(Level.DEBUG)

As an alternative, if you want to specify the level in a properties file, create a log4j.properties file and configure your own logger and set its properties in that file. For example:

# log4j.properties file (a text file)
log4j.rootLogger=DEBUG, STDOUT
log4j.appender.STDOUT=org.apache.log4j.ConsoleAppender
log4j.appender.STDOUT.layout=org.apache.log4j.PatternLayout
log4j.appender.STDOUT.layout.ConversionPattern=[%t] %5p (%F:%L) - %m%n

Put this file in your classpath. If you are using a Maven directory layout, put the file in the src/main/resources/ directory.

java.lang.OutOfMemoryError Exceptions

If a java.lang.OutOfMemoryError exception is thrown, increase the maximum heap size for the JVM.

If you are using the Scala REPL and you need to increase the maximum heap size, edit the run.sh shell script (provided in the archive file) and add the -J-Xmxmaximum_size flag to the scala command. The following example increases the maximum heap size to 4 GB:

scala -J-Xmx4G ...