I am trying to parse date using to_date() but I get the following exception. SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to parse '12/1/2010 8:26'...

I'm trying to run an example from the Spark book Spark: The Definitive Guide build.sbt ThisBuild / scalaVersion := "3.2.1" libraryDependencies ++= Seq( ("org.apache.spark" %% ...

In my spark job, I tried to overwrite a table in each microbatch of structured streaming batchDF.write.mode(SaveMode.Overwrite).saveAsTable("mytable") It generated the following error. ...

I'm using Spark 3.0.1 with user provided Hadoop 3.2.0 and Scala 2.12.10 running on Kubernetes. Everything works fine when reading a parquet file compressed as snappy, however when I try to read a p...

I have large amount of json files that Spark can read in 36 seconds but Spark 3.0 takes almost 33 minutes to read the same. On closer analysis, looks like Spark 3.0 is choosing different DAG than S...