How to Replicate an IF Then ELSE Statement in Apache Spark?

Spark Equivalent of IF Then ELSE

In Spark, you can apply conditional expressions to columns using the when() function. This function allows you to specify true and false values for different conditions.

Code Error and Solution

Your code throws an error because you are incorrectly using the when() function. The correct syntax for when() is:

when(condition, value).when(...)

or

when(condition, value).otherwise(...)

In your code, you have provided three arguments to the when() function, which is incorrect. To fix this, you need to rewrite your code as follows:

iris_spark_df = iris_spark.withColumn(
    "Class", 
    F.when(iris_spark.iris_class == 'Iris-setosa', 0)
    .when(iris_spark.iris_class == 'Iris-versicolor', 1)
    .otherwise(2)
)

Equivalent SQL Expression

The Spark when() function is equivalent to the CASE statement in SQL:

CASE
    WHEN (iris_class = 'Iris-setosa') THEN 0
    WHEN (iris_class = 'Iris-versicolor') THEN 1
    ELSE 2
END

The above is the detailed content of How to Replicate an IF Then ELSE Statement in Apache Spark?. For more information, please follow other related articles on the PHP Chinese website!