How to Implement \'IF THEN ELSE\' Logic in Spark DataFrames?

Spark Equivalent of "IF Then ELSE"

Introduction:

Spark DataFrame transformations offer a powerful way to manipulate data. One common operation involves conditionally applying transformations based on variable values. Understanding the Spark equivalent of the "IF Then ELSE" statement in SQL is crucial for such tasks.

Question:

A user is attempting to add a new column to a Spark DataFrame based on conditional rules. However, they encounter a TypeError when trying to use the F.when function with multiple conditions.

TypeError: when() takes exactly 2 arguments (3 given)

Answer:

The error occurs because the F.when function in Spark expects exactly two arguments: a condition and a value to return when the condition is met. The user's code includes an additional argument, another F.when condition, which is incorrect syntax.

The correct syntax for the "IF Then ELSE" equivalent in Spark using F.when is either:

(when(col("iris_class") == 'Iris-setosa', 0)
.when(col("iris_class") == 'Iris-versicolor', 1)
.otherwise(2))

or:

(when(col("iris_class") == 'Iris-setosa', 0)
    .otherwise(when(col("iris_class") == 'Iris-versicolor', 1)
        .otherwise(2)))

The first syntax uses nested F.when conditions, while the second uses the F.otherwise function.

An equivalent SQL statement would be a CASE expression:

CASE WHEN (iris_class = 'Iris-setosa') THEN 0 
     ELSE CASE WHEN (iris_class = 'Iris-versicolor') THEN 1 
               ELSE 2 
          END 
END

Spark also supports the Hive IF conditional syntax, but only in raw SQL with Hive support:

IF(condition, if-true, if-false)

The above is the detailed content of How to Implement \'IF THEN ELSE\' Logic in Spark DataFrames?. For more information, please follow other related articles on the PHP Chinese website!