I am trying to register a simple UDF for extracting date functionality in spark using Scala Luna Eclipse IDE.
This is my code:
sqlContext.udf.register("extract", (dateUnit: String, date : String) => udf.extract(dateUnit,date ) )
def extract(dateUnit : String, date: String) : String = {
val splitArray : Array[String] = date.split("-")
val result = dateUnit.toUpperCase() match {
case "YEAR" => splitArray(0)
case "MONTH" => splitArray(1)
case "DAY" => splitArray(2)
case whoa => "Unexpected case :" + whoa.toString()
}
return result ;
}
When I execute this functality an Eclipse console through
as Select * from date_dim WHERE d_dom < extract('YEAR', '2015-05-01') limit 10"
It throws ans error as
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:
Aggregate false, [], [Coalesce(SUM(PartialCount#30L),0) AS count#28L]
Aggregate true, [], [COUNT(1) AS PartialCount#30L]
Project []
Caused by: org.apache.spark.SparkException: Task not serializable
Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at java.io.ObjectStreamClass$FieldReflector.getObjFieldValues(ObjectStreamClass.java:2030)
at java.io.ObjectStreamClass.getObjFieldValues(ObjectStreamClass.java:1232)
I am not able to find out what exactly is the issue, the simple udfs that are defined directly like sqlContext.udf.register("strLength", (str: String) => str.length() )
runs successfully. And the same above function runs successfully through Scala shell in spark. What is the problem here. Am I doing any thing wrong?