Running shell script with Oozie

C

9

13

I am trying to run a sh script through Oozie, but I am facing a problem:

Cannot run program "script.sh" (in directory "/mapred/local/taskTracker/dell/jobcache/job_201312061003_0001/attempt_201312061003_0001_m_000000_0/work"): java.io.IOException: error=2, No such file or directory.

Please help me with necessary steps.

Craggie answered 6/12, 2013 at 4:32 Comment(3)

Step 1: does it exist? – Certified 6/12, 2013 at 4:34

run ls -la from oozie to prove that it's there. Did you copy it there manually of using oozie via <file>script.sh</file>? – Leitman 6/12, 2013 at 22:57

This is similar to https://mcmap.net/q/906409/-oozie-shell-script-action Good explanation given. – Melodymeloid 24/8, 2016 at 4:53

I

7

This error is really ambiguous. Here are some issues that have helped me to solve this issue.

-If you are running oozie workflows on a kerberized cluster, make sure to authenticate by passing your Kerberos Keytab as a argument:

...
<shell>
  <exec>scriptPath.sh</exec>
  <file>scriptPath.sh</file>
  <file>yourKeytabFilePath</file>
</shell>
...

-In your shell File (scriptPath.sh), make sure ro remove first line shell reference.

#!usr/bin/bash

indeed, if this shell reference isn't deployed on all data nodes, this can lead to this error code.

Intercourse answered 3/5, 2016 at 12:12 Comment(1)

Removing #!usr/bin/bash saved my life – Unsnarl 24/8, 2016 at 12:24

T

4

I had this same issue because of something really silly. I added a shell block in the workflow, then I selected the corresponding sendMail.sh, but I forgot to add the file sendMail.sh in FILE +.

Thicken answered 15/2, 2018 at 6:47 Comment(0)

L

1

workflow.xml :

...
<shell>
  <exec>script.sh</exec>

  <file>scripts/script.sh</file>
</shell>
...

Make sure you have scripts/script.sh in the same folder in hdfs.

Leitman answered 14/12, 2013 at 23:32 Comment(0)

P

1

An Oozie shell action is executed on a random Hadoop node, i.e. not locally on the machine where the Oozie server is running. As Oleksii says, you have to make sure that your script is on the node that executes the job.

See the following complete examples of executing a shell action and an ssh action:

https://github.com/airawat/OozieSamples/tree/master/oozieProject/workflowShellAction https://github.com/airawat/OozieSamples/tree/master/oozieProject/workflowSshAction

Pudens answered 15/12, 2013 at 19:49 Comment(0)

G

1

Try to give full path for HDFS like

<exec>/user/nathalok/run.sh</exec>  
<file>/user/nathalok/run.sh#run.sh</file>

and ensure that in job.properties the path is mentioned correctly for the library and workflow.xml

oozie.libpath=hdfs://server/user/oozie/share/lib/lib_20150312161328/oozie
oozie.wf.application.path=hdfs://bcarddev/user/budaledi/Teradata_Flow

Gelatin answered 7/5, 2015 at 4:37 Comment(1)

so your line should look like below: <exec>/hdfs_path/script.sh</exec> <file>/hdfs_path/scripts#script.sh</file> ensure you give # in second line – Gelatin 7/5, 2015 at 4:38

C

1

In addition to what others said, this can also be caused by a shell script having wrong line endings (e.g. CRLF for Windows). At least this happened to me :)

Caseycash answered 11/5, 2015 at 12:11 Comment(0)

A

1

if your shell file exist in your project correlated dir. then it's your shell file format cause this error. you need to convert the format from dos to linux using dos2linux:dos2linux xxxx.sh

Acreage answered 30/6, 2017 at 7:53 Comment(1)

Thanks. It helped me. I had to use set ff=unix on the script file. – Anesthetize 18/5, 2018 at 10:30

I

0

Also make sure that the shell scripts are UNIX compliant. If these shell scripts were written in windows environment then it appends windows specific end of lines (EOL) and these scripts are not recognized by the oozie. So you will get "no such file or directory found" in oozie shell actions.

Ial answered 8/12, 2016 at 10:56 Comment(0)

F

0

workflow.xml would look something like this

<workflow-app name="HiveQuery_execution" xmlns="uri:oozie:workflow:0.5">
<start to="shell-3c43"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="shell-3c43">
    <shell xmlns="uri:oozie:shell-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <exec>/user/path/hivequery.sh</exec>
        <file>/user/path/hivequery.sh#hivequery.sh</file>
          <capture-output/>
    </shell>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<end name="End"/>

Job.properties

jobTracker=xxxx.xxx.xxx.com:port
nameNode=hdfs://xxxx.xxx.xxx.com:port

better configure through UI, as suggested above

Framing answered 8/7, 2020 at 10:41 Comment(0)

Recommended topics

Hot tags