yarn application -list: How to filter list for today or last 24 hours?

Question

I am trying to list the applications run on Hadoop cluster. I can get the list to filter by application status as follows:

>yarn application -list -appStates FINISHED

But that still pulls up whole history (last 4-5 days, I guess based on Yarn Timeline server config).

Is there a way to filter that by a specific date or something like last 24 hours?

Nominee

Answer 1

You can use the RM Apps API to do this. For a simple test you can run:

$ date +"%s"
1495215569
$ let x=1495215569-86400
$ echo $x
1495129169
$ curl 'RMURL/ws/v1/cluster/apps?startedTimeBegin=1495129169000' | python -m json.tool

This pulls the apps that started when date was run minus one day (86400 seconds) and displays them. You need to add 000 as the time parameters take milliseconds not seconds. Supported parameters are:

• startedTimeBegin - applications with start time beginning with this time, specified in ms since epoch
• startedTimeEnd - applications with start time ending with this time, specified in ms since epoch
• finishedTimeBegin - applications with finish time beginning with this time, specified in ms since epoch
• finishedTimeEnd - applications with finish time ending with this time, specified in ms since epoch

See https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API for more details.

Gratia

Awesome @tk421. With bit more modification, this is how I plan to list all SPARK and SUCCEEDED applications for last 24 hours: curl "http://atl-hdp-022.afy.gbl:8088/ws/v1/cluster/apps?startedTimeBegin=date +%s%3N -d '1 day ago'&startedTimeEnd=date +%s%3N&applicationTypes=SPARK&finalStatus=SUCCEEDED" – Nominee