Installing and Configuring the Spark Connector¶

Step 1: Download the Latest Version of the Snowflake Connector for Spark¶

Snowflake provides multiple versions of the connector. Download the appropriate version, based on the following:

• The version of the Snowflake Connector for Spark that you want to use.

• The version of Spark that you are using.

• The version of Scala that you are using.

You can download the Snowflake Spark Connector from Maven. If you want to build the driver, you can access the source code from Github.

Step 2: Download the Compatible Version of the Snowflake JDBC Driver¶

Next, you need to download the version of the Snowflake JDBC driver that is compatible with the version of the Snowflake Spark Connector that you are using.

The Snowflake JDBC driver is provided as a standard Java package through the JDBC Driver page in the Maven Central Repository. You can either download the package as a .jar file or you can directly reference the package. These instructions assume you are referencing the package.

To find the supported version of the Snowflake JDBC Driver for the version of the Snowflake Spark Connector that you are using, see the Snowflake Connector for Spark release notes.

For more details on downloading and installing the Snowflake JDBC Driver, see Downloading / integrating the JDBC Driver.

Step 3 (Optional): Verify the Snowflake Connector for Spark Package Signature¶

To verify the Snowflake Connector for Spark package signature:

1. From the public keyserver, download and import the Snowflake GPG public key for the version of the Snowflake Connector for Spark that you are using:

   • For version 2.11.1 and higher:

     $ gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 630D9F3CAB551AF3

   • For version 2.8.2 through 2.11.0:

     $ gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 37C7086698CB005C

   • For version 2.4.13 through 2.8.1:

     $ gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys EC218558EABB25A1

   • For version 2.4.12 and lower:

     $ gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 93DB296A69BE019A

   Note

   If this command fails with the following error:

   gpg: keyserver receive failed: Server indicated a failure
   

   Copy

   then specify that you want to use port 80 for the keyserver:

   gpg --keyserver hkp://keyserver.ubuntu.com:80  ...
   

   Copy

2. Run the gpg --verify command to verify the signature of the package.

   For the --verify command-line flag, specify the .asc file that you downloaded earlier as the signature file and the JAR file as the file containing the signed data.

   $ gpg --verify spark-snowflake_x.xx-N.N.N-spark_P.P.jar.asc spark-snowflake_x.xx-N.N.N-spark_P.P.jar
   gpg: Signature made Wed 22 Feb 2017 04:31:58 PM UTC using RSA key ID <gpg_key_id>
   gpg: Good signature from "Snowflake Computing <snowflake_gpg\ @snowflake.net>"
   

   Copy

   where:

   • x.xx is the Scala version (e.g. 2.12).

   • N.N.N is the version of the Snowflake Connector for Spark (e.g. 2.12.0).

   • P.P is the Spark version (e.g. 3.4).

   Note

   Verifying the signature produces a warning similar to the following:

   gpg: Signature made Mon 24 Sep 2018 03:03:45 AM UTC using RSA key ID <gpg_key_id>
   gpg: Good signature from "Snowflake Computing <snowflake_gpg@snowflake.net>" unknown
   gpg: WARNING: This key is not certified with a trusted signature!
   gpg: There is no indication that the signature belongs to the owner.
   

   Copy

   To avoid the warning, you can grant the Snowflake GPG public key implicit trust.

3. Your local environment can contain multiple GPG keys; however, for security reasons, Snowflake periodically rotates the public GPG key. As a best practice, we recommend deleting the existing public key after confirming that the latest key works with the latest signed package. For example:

   $ gpg --delete-key "Snowflake Computing"
   

   Copy

Step 4: Configure the Local Spark Cluster or Amazon EMR-hosted Spark Environment¶

If you have a local Spark installation, or a Spark installation in Amazon EMR, you need to configure the spark-shell program to include both the Snowflake JDBC driver and the Spark Connector:

• To include the Snowflake JDBC driver, use the --package option to reference the JDBC package from the JDBC Driver page in the Maven Central Repository, providing the exact version of the driver you wish to use (e.g. net.snowflake:snowflake-jdbc:3.13.30).

• To include the Spark Connector, use the --package option to reference the appropriate package (Scala 2.12 or Scala 2.13) hosted in the Maven Central Repository, providing the exact version of the driver you want to use (e.g. net.snowflake:spark-snowflake_2.12:2.12.0-spark_3.4).

For example:

spark-shell --packages net.snowflake:snowflake-jdbc:3.13.22,net.snowflake:spark-snowflake_2.12:2.12.0-spark_3.4

Copy

Preparing an AWS External S3 Bucket¶

Prepare an external S3 bucket that the connector can use to exchange data between Snowflake and Spark. You then provide the location information, together with the necessary AWS credentials for the location, to the connector. For more details, see Authenticating S3 for Data Exchange in the next topic.

Preparing an Azure Blob Storage Container¶

Prepare an external Azure Blob storage container that the connector can use to exchange data between Snowflake and Spark. You then provide the location information, together with the necessary Azure credentials for the location, to the connector. For more details, see Authenticating Azure for Data Exchange in the next topic.