Snowpark Migration Accelerator: Snowpark Connect Issue Codes for Python¶

Description¶

This issue appears when the tool detects the usage of an element that is not supported in Snowpark Connect, and does not have its own error code associated with it. This is the generic error code used by the SMA for an unsupported element.

Scenario¶

from pyspark import NotSupportedElement
sc.addFile("data.txt")
print(NotSupportedElement.get("data.txt"))

from pyspark import NotSupportedElement
sc.addFile("data.txt")
# EWI SPRKCNT1000: The element 'NotSupportedElement' is not supported for Snowpark Connect
print(NotSupportedElement.get("data.txt"))

Recommended fix¶

Since this is a generic error code that applies to a range of unsupported functions, there is not a single and specific fix. The appropriate action will depend on the particular element in use.

Additional recommendations¶

• Even though the option or the element on the message is not supported, this does not mean that a solution cannot be found. It means only that the tool itself cannot find the solution.

• If you believe that Snowpark Connect already supports this element or that there is some kind of workaround, please report that you encountered a conversion error on that particular element using the Report an Issue option in the SMA and include any additional information that you think may be helpful.

• For more support, email support at sma-support@snowflake.com or post an issue in the SMA.

Description¶

This issue appears when the tool determines that the usage instance of an RDD element is not supported in Snowpark Connect, and does not have its own error code associated with it. This is the generic error code used by the SMA for RDD unsupported elements.

Scenario¶

from pyspark import SparkContext

sc = SparkContext("local", "Simple App")
data = [1, 2, 3, 4, 5]
rdd = sc.parallelize(data)
result = rdd.NotSupportedElement(withReplacement=False, fraction=0.5).collect()
print(result)

from pyspark import SparkContext

sc = SparkContext("local", "Simple App")
data = [1, 2, 3, 4, 5]
rdd = sc.parallelize(data)
# EWI SPRKCNT1500: The element 'NotSupportedElement' of the library RDD is not supported for Snowpark Connect
result = rdd.NotSupportedElement(withReplacement=False, fraction=0.5).collect()
print(result)

Recommended fix¶

• Convert RDD operations to DataFrame operations.

  # PySpark RDD approach (NOT supported)
  rdd = spark.sparkContext.parallelize(data)
  result = rdd.map(lambda x: x * 2).collect()
  
  # Snowpark DataFrame approach (Supported)
  df = session.create_dataframe(data, schema=["value"])
  result = df.select(col("value") * 2).collect()
  

  Copy

• Process data locally before sending it to Snowflake if the RDD operations are simple.

• Use pandas DataFrames for local processing, then convert to Snowpark DataFrames.

• Since this is a generic error code that applies to a range of unsupported functions, there is no single and specific fix. The appropriate action will depend on the particular element in use.

• Please note that even though the element is not supported, it does not necessarily mean that a solution or workaround cannot be found. It means only that the SMA itself cannot find the solution.

Additional recommendations¶

• Even though the option or the element on the message is not supported, this does not mean that a solution cannot be found. It means only that the tool itself cannot find the solution.

• If you believe that Snowpark Connect already supports this element or that there is a workaround, please report that you encountered a conversion error on that particular element using the Report an Issue option in the SMA and include any additional information that you think may be helpful.

• For more support, email support at sma-support@snowflake.com or post an issue in the SMA.

Description¶

This issue appears when the tool determines that the usage instance of a Streaming element is not supported in Snowpark Connect and does not have its own error code associated with it. This is the generic error code used by the SMA for Streaming unsupported elements.

Scenario¶

from pyspark import SparkContext
from pyspark.streaming import NotSupportedElement

sc = SparkContext("local[2]", "NetworkWordCount")
ssc = NotSupportedElement(sc, 1)

from pyspark import SparkContext
from pyspark.streaming import NotSupportedElement

sc = SparkContext("local[2]", "NetworkWordCount")
# EWI SPRKCNT2000: The element 'NotSupportedElement' of the library Streaming is not supported for Snowpark Connect
ssc = NotSupportedElement(sc, 1)

Recommended fix¶

• Use Snowflake Streams to capture table changes.

• Process changes with Snowpark Connect in scheduled jobs.

• Ideal for maintaining derived tables.

• Since this is a generic error code that applies to a range of unsupported functions, there no single and specific fix. The appropriate action will depend on the particular element in use.

• Please note that even though the element is not supported, it does not necessarily mean that a solution or workaround cannot be found. It means only that the SMA itself cannot find the solution.

Additional recommendations¶

• Even though the option or the element on the message is not supported, this does not mean that a solution cannot be found. It means only that the tool itself cannot find the solution.

• If you believe that Snowpark Connect already supports this element or that there is a workaround, please report that you encountered a conversion error on that particular element using the Report an Issue option in the SMA and include any additional information that you think may be helpful.

• For more support, email support at sma-support@snowflake.com or post an issue in the SMA.

Description¶

This issue appears when the tool determines the usage instance of an ML element that is not supported in Snowpark Connect and does not have its own error code associated with it. This is the generic error code used by the SMA for ML unsupported elements.

Scenario¶

from pyspark.ml.classification import NotSupportedElement

lr = NotSupportedElement(maxIter=10, regParam=0.01)
model = lr.fit(trainingData)

from pyspark.ml.classification import NotSupportedElement

# EWI SPRKCNT2500: The element 'NotSupportedElement' of the library ML is not supported for Snowpark Connect
lr = NotSupportedElement(maxIter=10, regParam=0.01)
model = lr.fit(trainingData)

Recommended fix¶

• Use the Snowpark ML library.

  # Instead of PySpark ML
  from snowflake.ml.modeling.linear_model import LinearRegression
  from snowflake.ml.modeling.ensemble import RandomForestRegressor
  
  # Snowpark ML approach
  model = LinearRegression()
  model.fit(train_df)
  predictions = model.predict(test_df)
  

  Copy

• Since this is a generic error code that applies to a range of unsupported functions, there is no single and specific fix. The appropriate action will depend on the particular element in use.

• Please note that even though the element is not supported, it does not necessarily mean that a solution or workaround cannot be found. It means only that the SMA itself cannot find the solution.

Additional recommendations¶

• Even though the option or the element on the message is not supported, this does not mean that a solution cannot be found. It means only that the tool itself cannot find the solution.

• If you believe that Snowpark Connect already supports this element or that there is some kind of workaround, please report that you encountered a conversion error on that particular element using the Report an Issue option in the SMA and include any additional information that you think may be helpful.

• For more support, email support at sma-support@snowflake.com or post an issue in the SMA.

Description¶

This issue appears when the tool determines the usage instance of an MLLIB element that is not supported in Snowpark Connect, and does not have its own error code associated with it. This is the generic error code used by the SMA for ML unsupported elements.

Scenario¶

from pyspark.mllib.recommendation import NotSupportedElement, Rating
ratings = [
    Rating(0, 0, 4.0),
    Rating(0, 1, 2.0),
    Rating(1, 1, 5.0)
]
model = NotSupportedElement.train(ratings, 10)

from pyspark.mllib.recommendation import NotSupportedElement, Rating

# EWI SPRKCNT3000: The element 'NotSupportedElement' of the library MLLIB is not supported for Snowpark Connect
ratings = [
    Rating(0, 0, 4.0),
    Rating(0, 1, 2.0),
    Rating(1, 1, 5.0)
]
model = NotSupportedElement.train(ratings, 10)

Recommended fix¶

• Use the Snowpark ML library.

  # Instead of MLlib's LinearRegressionWithSGD
  from snowflake.ml.modeling.linear_model import LinearRegression
  
  # MLlib approach (NOT supported)
  # from pyspark.mllib.regression import LinearRegressionWithSGD
  # model = LinearRegressionWithSGD.train(rdd_data)
  
  # Snowpark ML approach
  model = LinearRegression(input_cols=['feature1', 'feature2'], label_cols=['target'])
  model.fit(train_df)
  

  Copy

• Since this is a generic error code that applies to a range of unsupported functions, there is no single and specific fix. The appropriate action will depend on the particular element in use.

• Please note that even though the element is not supported, it does not necessarily mean that a solution or workaround cannot be found. It means only that the SMA itself cannot find the solution.

Additional recommendations¶

• Even though the option or the element on the message is not supported, this does not mean that a solution cannot be found. It means only that the tool itself cannot find the solution.

• If you believe that Snowpark Connect already supports this element or that there is a workaround, please report that you encountered a conversion error on that particular element using the Report an Issue option in the SMA and include any additional information that you think may be helpful.

• For more support, email support at sma-support@snowflake.com or post an issue in the SMA.

Description¶

This issue appears when the tool determines that the usage instance of a Spark Session element is not supported in Snowpark Connect and does not have its own error code associated with it. This is the generic error code used by the SMA for ML unsupported elements.

Scenario¶

from pyspark.sql import SparkSession
from pyspark.sql.SparkSession import NotSupportedElement

spark = SparkSession.builder.appName("Example").getOrCreate()
new_spark = spark.NotSupportedElement()

from pyspark.sql import SparkSession
from pyspark.sql.SparkSession import NotSupportedElement

spark = SparkSession.builder.appName("Example").getOrCreate()
# EWI SPRKCNT3500: The element 'NotSupportedElement' of the library Spark Session is not supported for Snowpark Connect
new_spark = spark.NotSupportedElement()

Recommended fix¶

• The key is to replace SparkSession patterns with equivalent Snowpark Session operations while leveraging Snowflake's unique capabilities like warehouses, stages, and native SQL functions.

  # PySpark SparkSession
  from pyspark.sql import SparkSession
  spark = SparkSession.builder.appName("MyApp").getOrCreate()
  
  # Snowpark Session
  from snowflake.snowpark import Session
  session = Session.builder.configs({
      "account": "your_account",
      "user": "your_user", 
      "password": "your_password",
      "role": "your_role",
      "warehouse": "your_warehouse",
      "database": "your_database",
      "schema": "your_schema"
  }).create()
  

  Copy

• Since this is a generic error code that applies to a range of unsupported functions, there is no single and specific fix. The appropriate action will depend on the particular element in use.

• Please note that even though the element is not supported, it does not necessarily mean that a solution or workaround cannot be found. It means only that the SMA itself cannot find the solution.

Additional recommendations¶

• Even though the option or the element on the message is not supported, this does not mean that a solution cannot be found. It means only that the tool itself cannot find the solution.

• If you believe that Snowpark Connect already supports this element or that there is some kind of workaround, please report that you encountered a conversion error on that particular element using the Report an Issue option in the SMA and include any additional information that you think may be helpful.

• For more support, email support at sma-support@snowflake.com or post an issue in the SMA.