Pyspark Function to Negate Column

Question

Is there a built-in function to add a new column which is the negation of the original column?

Spark SQL has the function negative(). Pyspark does not seem to have inherited this function.

df_new = df.withColumn(negative("orginal"))

Directorate

I think you are asking: ``` df_new = df.withColumn("new_column_name", [what put in here?]) ``` – Avery

Answer 1

Assuming your column original is boolean :

df_new = df.withColumn(~df["original"])  # Equivalent to "not original"

Lais

Thanks Pierre, it looks like the '~' operator only works on Boolean Types. This operator is handy though – Directorate

Is there something pyspark native to do this? I think if I use ~ then it is going to be executed in a python process using memory overhead, not in the JVM. Is there an alternative that will be executed within the JVM only? – Nicotiana

Answer 2

I think it should be this to be syntax right, based on @pierre-gourseaud's answer:

df_new = df.withColumn("new_column_name", ~df["original"])  # Equivalent to "not original"

Avery