'Unexpected end of file' and 'error importing function definition' error running shellscript using qsub

Question

I have the following shellscript:

#!/bin/sh
cd /sw/local/bin/
export LD_LIBRARY_PATH=/sw/local/lib:/usr/local/Trolltech/Qt-4.7.2/lib:$LD_LIBRARY_PATH
./FeatureFinderRaw -in /homes/JG-C1-18.mzML -out /homes/test_remove_after_use.featureXML -threads 20

It works fine when I run it from my own command line, but when I try to do:

qsub -q ningal.q -cwd -V -o /homes/queue.out -e /queue.err featureFind_C1-18_20t.sh 

I get the following error:

/bin/sh: module: line 1: syntax error: unexpected end of file
/bin/sh: error importing function definition for `module'
./FeatureFinderRaw: error while loading shared libraries: libOpenMS.so: cannot open shared object file: No such file or directory
/bin/sh: module: line 1: syntax error: unexpected end of file
/bin/sh: error importing function definition for `module'
./FeatureFinderRaw: error while loading shared libraries: libQtWebKit.so.4: cannot open shared object file: No such file or directory
/bin/sh: module: line 1: syntax error: unexpected end of file
/bin/sh: error importing function definition for `module'
./FeatureFinderRaw: error while loading shared libraries: libQtWebKit.so.4: cannot open shared object file: No such file or directory
/bin/bash: module: line 1: syntax error: unexpected end of file
/bin/bash: error importing function definition for `module'
./FeatureFinderRaw: error while loading shared libraries: libQtWebKit.so.4: cannot open shared object file: No such file or directory
/bin/sh: module: line 1: syntax error: unexpected end of file
/bin/sh: error importing function definition for `module'
./FeatureFinderRaw: error while loading shared libraries: libQtWebKit.so.4: cannot open shared object file: No such file or directory
/bin/sh: module: line 1: syntax error: unexpected end of file
/bin/sh: error importing function definition for `module'

I don't understand why I get this error when using qsub, but not when running the script directly on the same cluster machine. How can I run a script using qsub?

Piloting

Answer 1

Also have this problem in a wrapper script that uses

qsub -shell no -b yes -cwd -V somescript.bash arg1 arg2 etc

if you use it to submit another bash shell script. It produces the annonying

/bin/sh: module: line 1: syntax error: unexpected end of file
/bin/sh: error importing function definition for `BASH_FUNC_module'

(this is Sun Grid Engine 211.11 running on CentOS 6.6) Turns out things are solved by simply putting the following on top of the wrapper script (not of the wrapped script):

unset module

That's all.

Calculation

Answer 2

In /usr/share/Modules/init/bash commented out the 'export -f module' line.

In a normal login shell, modules.sh will be called from profile.d so the module command is available. In a non login shell, like an app wrapper script it just source the above file first.

Generally in applications script after sourcing above file they again give command "module load apps/vendor/app" which means extra sourcing.

Reference::- http://gridengine.org/pipermail/users/2011-November/002019.html

Cayser

Answer 3

For some reason unknown to me adding semicolons at the end of every line fixed the problem.

Piloting

At every line of /usr/share/Modules/init/bash? (Having the same problem here). – Missionary

No at the end of the shellscript – Piloting

Answer 4

Add the command below to ~/.bash_profile or ~/.bashrc file then logout/login again.

unset module

Or simple run quick command:

echo "unset module" >> ~/.bash_profile && source ~/.bash_profile

Beffrey

Answer 5

Most likely cause is that you saved the file with DOS line endings (\r\n) instead of POSIX line endings (\r) if adding semicolons make a difference.

Corene

I ran dos2unix, that didn't make a difference. Also, I wrote it on linux and it works when I run it on the same linux cluster without using qsub. – Piloting

Answer 6

confluence.oraclecorp.com/confluence/pages/viewpage.action?pageId=3567665781 will help

The following can be used as work around.

Adding the following to the $AW_HOME/site/sosite file. Once added, the Applications Manager process will need to be stopped, and sosite needs to be run before before restarting for the change to take effect.

unset module unset _module_raw unset switchml unset ml unset scl

This is scheduled for a permanent fix in a future release of Applications Manager

Assist