How can I control the Perl version used when submitting grid jobs?

Question

I'm working with SGE (Sun Grid Engine) to submit jobs to a grid. I also use perlbrew to manage my installed Perl versions. I wrote some short sh scripts that I use to run a perl script which requires a specific Perl version (5.12.2), which look something like this:

#!/bin/bash
#$-S /bin/bash

source /home/dave/.bash_profile
/home/dave/perl5/perlbrew/bin/perlbrew switch perl-5.12.2

/home/dave/scripts/proc_12.pl --in=/home/dave/in/in.store --dir=/home/dave/in/dir2 --params=/home/dave/in/params.p

Now, when I submit a single job everything works fine, but when I submit many, I start getting perlbrew related error messages, like:

ln: creating symbolic link `current' to `perl-5.12.2': File exists
ln: creating symbolic link `/home/dave/perl5/perlbrew/bin/cpan' to `/home/dave/perl5/perlbrew/perls/current/bin/cpan': File exists
ln: creating symbolic link `/home/dave/perl5/perlbrew/bin/cpan2dist' to `/home/dave/perl5/perlbrew/perls/current/bin/cpan2dist': File exists
ln: cannot remove `/home/dave/perl5/perlbrew/bin/cpanp': No such file or directory
ln: cannot remove `/home/dave/perl5/perlbrew/bin/enc2xs': No such file or directory
ln: cannot remove `/home/dave/perl5/perlbrew/bin/find2perl': No such file or directory

So I guess the /home/dave/perl5/perlbrew/bin/perlbrew switch perl-5.12.2 line is causing the problems.

What can I do?

How can I make my script run using perl-5.12.2 (the default is 5.8.8)?

Insurrectionary

Apart from the "don't use perlbrew for this" answer others have, this does look a bit like an issue that could be solved with locking during the perlbrew run. Since this is in a cluster, you might be interested in <a href="search.cpan.org/…>. Caveat: I wrote it. – Eventuality

Answer 1

I don't recommend putting the perlbrew switch perl-5.12.2 in any script you run. Its really only for command line usage.

If you need a script to use a specific version of Perl then either give it the full perlbrew path on the shebang:

#!/home/dave/perl5/perlbrew/perls/perl-5.12.2/bin/perl

use 5.012;
use warnings;
...

Then make sure its executable and run like so:

chmod +x your_perl_program.pl
./your_perl_program.pl

Or alternatively use the full pathname to the perl binary in your script:

#!/bin/bash

/home/dave/perl5/perlbrew/perls/perl-5.12.2/bin/perl your_perl_program.pl

BTW, you will have potential production and security issues if you run anything unqualified in your scripts or perl programs. For eg:

#!/bin/sh

# security risk
perl some_script.pl

# and not just perl
tar cvf archive.tar *.txt

# production risk
/home/dave/perl5/perlbrew/bin/perl some_other_script.pl

The first two are bad because it will pick up the first perl & tar it finds in your path. So this depends on $PATH setting and this could become a security risk. The last is also not good because its reliant on what perl perlbrew is currently switched to at the point in time its run :(

So doing this can be a potential production & security nightmare. Instead the above should be written like this:

#!/bin/sh

# fully qualified now.  Uses OS provided perl
/usr/bin/perl some_script.pl

# ditto
/usr/bin/tar cvf archive.tar *.txt

# this needs to run in perl 5.12.2
/home/dave/perl5/perlbrew/perls/perl-5.12.2/bin/perl some_other_script.pl

Hope that all makes sense?

Bilski

Please see my reply to brian: #3776890. Namely, will scripts called within this script will also use the same perl version or resort to the default? – Insurrectionary

If you qualify every script you use like I explained above then all will be OK. if you don't then it opens up a can of worms! (see my BTW). – Bilski

OK. So when I call a script from within perl, I also need to explicitly pick the perl I wish to use (i.e. system($my_perl, $my_script,@args), right? – Insurrectionary

Correct... and also for any other binary you run from system (see my tar update). – Bilski

Answer 2

I'd recommend against using perlbrew. It doesn't really give you that much value and it just causes confusion about what perl you are using. perlbrew really assumes that everything running at the same time agrees on which perl they should use. I think that's just a recipe for headaches as different programs start switching perls out from under you, perhaps before you get the chance even to use the perl you think you switched to.

Just install the perls that you want and call the one that you want.

 $ perl5.12.2 /home/dave/scripts/proc_12.pl ...

For instance, to install a perl you just run from the source tree (with whichever prefix you want):

 $ ./Configure -des -Dprefix=/usr/local/perls/perl-5.12.2
 $ make install

I then make symlinks to all my installed perls with my make_links script. When I want to use Perl 5.12.2, I just use ~/bin/perl5.12.2. I never have to switch perls. When I want to install a Perl module, I use the cpan for it:

 cpan5.12.2 Some::Module

I've never unsure about what version I'm using, and there's no race condition in moving symlinks around.

Mcgruter

Thank you brian. I know start to understand why you have nothing against perlbrew but still thinks it's better not use it. I'll think I'll drop it since it causing more overheads the benefits. So suppose I will now call my perl scripts using something like ~/bin/perl5.12.2, and suppose the perl script itself calls some other perl scripts (not mine), by simply invoking them using e.g. system. Will those other scripts run under 5.12.2 too? Also, does invoking a script using an explicit perl bin overrides the Shebang line in the script? – Insurrectionary

If you want to run the same perl, use the $^X variable inside your script. See perlvar. – Mcgruter