How to update a value in the nested column of struct using pyspark

Question

I try to do very simple - update a value of a nested column;however, I cannot figure out how

Environment:

1. Apache Spark 2.4.5
2. Databricks 6.4
3. Python 3.7

dataDF = [
  (('Jon','','Smith'),'1580-01-06','M',3000)
]


schema = StructType([
        StructField('name', StructType([
             StructField('firstname', StringType(), True),
             StructField('middlename', StringType(), True),
             StructField('lastname', StringType(), True)
             ])),
         StructField('dob', StringType(), True),
         StructField('gender', StringType(), True),
         StructField('gender', IntegerType(), True)
         ])


df = spark.createDataFrame(data = dataDF, schema = schema)
df = df.withColumn("name.firstname", lit('John'))
df.printSchema()
df.show()

#Results
#I get a new column instead of update

root
 |-- name: struct (nullable = true)
 |    |-- firstname: string (nullable = true)
 |    |-- middlename: string (nullable = true)
 |    |-- lastname: string (nullable = true)
 |-- dob: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- gender: integer (nullable = true)
 |-- name.firstname: string (nullable = false)

+--------------+----------+------+------+--------------+
|          name|       dob|gender|gender|name.firstname|
+--------------+----------+------+------+--------------+
|[Jon, , Smith]|1580-01-06|     M|  3000|          John|
+--------------+----------+------+------+--------------+

Theomania

Answer 1

Need to wrangle with the column a bit as below:

import pyspark.sql.functions as F

df2 = df.select('*', 'name.*') \
        .withColumn('firstname', F.lit('newname')) \
        .withColumn('name', F.struct(*[F.col(col) for col in df.select('name.*').columns])) \
        .drop(*df.select('name.*').columns)

df2.show()
+------------------+----------+------+------+
|              name|       dob|gender|gender|
+------------------+----------+------+------+
|[newname, , Smith]|1580-01-06|     M|  3000|
+------------------+----------+------+------+

Rayner

could you explain in more detail what .withColumn('name', F.struct([F.col(col) for col in df.select('name.').columns])) is doing please – Theomania

@Rayner can you add some descriptions to explain how this works? – Stereochemistry

@BIDude in this line (construction) is defining a new field with Struct type. – Wive

Answer 2

For Spark 3.1+, you can use withField on a struct column:

An expression that adds/replaces field in StructType by name.

import pyspark.sql.functions as F

df1 = df.withColumn("name", F.col("name").withField("firstname", F.lit("John")))

Astrix

Answer 3

The pyspark-nested-functions library allows for readable code:

from nestedfunctions.functions.terminal_operations import apply_terminal_operation
from pyspark.sql.functions import when
processed = apply_terminal_operation(
      df,
      field = "name.firstname",
      f = lambda x, type: when(x=='Jon', 'John').otherwise(x)
  )
processed.show()
# +---------------+----------+------+------+
# |           name|       dob|gender|gender|
# +---------------+----------+------+------+
# |{John, , Smith}|1580-01-06|     M|  3000|
# +---------------+----------+------+------+

Carbuncle