How to check whether the file exist in HDFS location, using oozie?

Question

How to check whether a file in HDFS location is exist or not, using Oozie?

In my HDFS location I will get a file like this test_08_01_2016.csv at 11PM , on a daily basis.

I want check whether this file exist after 11.15 PM. I can schedule the batch using a Oozie coordinator job.

But how can I validate if the file exists in HDFS?

Novanovaculite

Answer 1

you can use EL expression in oozie like:

<decision name="CheckFile">
         <switch>
            <case to="nextOozieTask">
              ${fs:exists('/path/test_08_01_2016.csv')} <!--do note the path which should be in ''-->
            </case>
            <default to="MailActionFileMissing" />
         </switch>
</decision>

You can also build the name of the file using simple shell script using capture output.

Staphylococcus

Hi Abhiieor, Thanks for ur reply.its working as expected :) ,i want to get today's date value dynamically to the file name i.e test_08_22_2016. i have tried coordination job to get the date value but it is not working..can please suggest on how can i get and append the date value in file. – Novanovaculite

Use #1401982, convert - to _ (say date1) and then export path='/path/test_'$date1'.csv' – Staphylococcus

thanks ,i have tried above date format it will work as expected for todays date .but in my case if todays file is not there means i.e (test_08_22_2016) , i have to verify last one week files (test_08_21_2016,test_08_20_2016...etc). in that case how can i use this – Novanovaculite

generate date for last one week and create file names and check in oozie decision. – Staphylococcus

sorry ,can you please elaborate answer. i have tried to generate date from oozie coordination job, i am able to get date value from the coordination job but i am not able to add/append date value in file name in oozie job. – Novanovaculite