How to get the current local time or system time in spark-scala dataframe?

Question

I am trying to get the local time in spark-scala but it is returning UTC.

I am using java.time.LocalDateTime to get the current timestamp. But its returning the UTC standard.

java.sql.Timestamp.valueOf(DateTimeFormatter.ofPattern("YYYY-MM-dd HH:mm:ss.SSSSSS").format(LocalDateTime.now))

The LocalDateTime is returning local time in spark shell, but in my code it is giving UTC standard.

val time: LocalDateTime = LocalDateTime.now

How to get the current time?

The current output is UTC. I need the local time. I need to change the zone.

Truda

I recommend you don’t use Timestamp. That class is poorly designed and long outdated. Instead use a class from java.time, the modern Java date and time API. – Expansible

Answer 1

The best way I know is to use GMT offset based on your timezone. Get your GMT offset and replace. My GMT offset is -5 in this example. Let me know if it helps. let me know if his solves the problem.

df=spark.sql("select from_utc_timestamp(current_timestamp(),'GMT-5') AS your_local_datetime")
df.show(truncate=False)

Stan

Answer 2

Use current_timestamp() in org.apache.spark.sql.fuctions, this gives local time.

Bastogne

Thank you, Yes I have tried that. But that is also returning UTC. Do you know how to config spark to local time? – Truda

if you want to retrieve as dataframe then spark.range(1).select(current_timestamp).show(false) otherwise you can retrive it by using first like this "var date_ = spark.range(1).select(current_timestamp).first.get(0)" – Saccule

2019-09-28 15:17:20.079 This is the time returned by this answer (UTC time). My local time is 20:48 now. – Truda