Running shell script from oozie through Hue

Question

I am invoking a bash shell script using oozie editor in Hue.

I used the shell action in the workflow and tried below different options in shell command:

1. Uploaded the shell script using 'choose a file'
2. Gave local directory path where shell script is present
3. Gave HDFS path where shell script is present

But all these options gave following error:

Cannot run program "sec_test_oozie.sh" (in directory "/data/hadoop/yarn/local/usercache/user/appcache/application_1399542362142_0086/container_1399542362142_0086_01_000002"): java.io.IOException: error=2, No such file or directory

How should I give the shell script execution command? Where the shell script file should be residing?

Debarath

Search for java.io.IOException: error=2, No such file or directory – Beatty

@CruncherBigData please give a look to SergioRG solution and the extension I wrote. The error is very confusing and gave me an headache. – Vogul

Answer 1

You need add file "sec_test_oozie.sh" in oozie shell step. In add files

Magnetron

The error is confusing itself , It says no such file or directory when we already uploaded the .sh file. – Conceptionconceptual

Answer 2

I think you are creating the file from windows machine which is adding extra line break characters.You need to convert the shell script file to Unix format.I also faced the same issue.Then I created the file from a Linux system and it started working.The error is misguiding.

Frazee

You are my solution. – Busk

Answer 3

I want to extend the @SergioRG answer. Oozie, at least with Cloudera's Hue interface is very counterintuitive.

To run a script file, three conditions should be met:

1. the file is on the HDFS file system, in a folder accessible by Oozie
2. the file should be indicated in the shell command field
3. the file should be added with any other dependent file in the "Files+" part of the task card.

I wonder why they didn't add by default the script file you are calling.

Edit: please also check in advanced options (the gear in the left upper corner) if you need to set the path variable (eg. PATH=/usr/local/bin:/usr/bin).

Vogul

Answer 4

Did you edit sec_test_oozie.sh with the Hue File Browser? Depending on your Hue version it might have corrupted it: hue-list

Robison

No Romain... I uploaded the Script file from local through 'Upload File' option as well as tried directly giving the HDFS path where shell script resides. – Debarath

Answer 5

I encountered the same problem, and the problem was that the script echoed some irrelevant line while the workflow tried to parse it as a property line. Oozie gave a very irrelevant error message of java.io.IOException: error=2, No such file or directory which only added confusion.

1. You will need to use <file> to add your script.
2. If you used <capture-output/> then you must make sure that your script prints only "key=value" lines, like java properties, otherwise you will get the error you see java.io.IOException: error=2, No such file or directory with some path pointing to .../yarn/local/usercache/...

Westerfield

Answer 6

We had this issue on a test script, basically if you use an editor that adds wierd characters or line ending to the file, it'll throw this error because the script cannot be used in the container.

Try using nano file.sh to see if any strange characters appear. Then push it back to hdfs with hdfs dfs -put file.sh /path/you/need

Herculaneum

Answer 7

"No such a file or directory" oozie cannot locate the file. Please check the AddPath setting in the command.

In the edit node seciton, get the oozie application hdfs path.

Upload the shell script in hdfs oozie application path.

In the oozie edit node step, Shell command - specify the shell script name which is uploaded.

Below that there would be option to AddPath, then add files, add the shell script which was uploaded in the hdfs path.

Myrtlemyrvyn

Answer 8

Removing the #!/bin/bash from my shell script helped me

Geoff