correct way to find scripts directory from setup.py in Python distutils?

Question

I am distributing a package that has this structure:

mymodule:
  mymodule/__init__.py
  mymodule/code.py
  scripts/script1.py
  scripts/script2.py

The mymodule subdir of mymodule contains code, and the scripts subdir contains scripts that should be executable by the user.

When describing a package installation in setup.py, I use:

scripts=['myscripts/script1.py']

To specify where scripts should go. During installation they typically go in some platform/user specific bin directory. The code that I have in mymodule/mymodule needs to make calls to the scripts though. What is the correct way to then find the full path to these scripts? Ideally they should be on the user's path at that point, so if I want to call them out from the shell, I should be able to do:

os.system('script1.py args')

But I want to call the script by its absolute path, and not rely on the platform specific bin directory being on the PATH, as in:

# get the directory where the scripts reside in current installation
scripts_dir = get_scripts_dir()
script1_path = os.path.join(scripts_dir, "script1.py")
os.system("%s args" %(script1_path))

How can this be done? thanks.

EDIT removing the code outside of a script is not a practical solution for me. the reason is that I distribute jobs to a cluster system and the way I usually do it is like this: imagine you have a set of tasks you want to run on. I have a script that takes all tasks as input and then calls another script, which runs only on the given task. Something like:

main.py:
for task in tasks:
  cmd = "python script.py %s" %(task)
  execute_on_system(cmd)

so main.py needs to know where script.py is, because it needs to be a command executable by execute_on_system.

Adjutant

Related:determining the location of distutils data files programmatically in python? – Slavic

How does execute_on_system() look like? – Slavic

@PiotrDobrogost: execute_on_system takes as input a string corresponding to a shell command. It then creates a temporary file called cmd.sh and puts in it the command, along with necessary prefixes that state how the job should run (specific to the cluster system being used on the user's system) -- eg what queue the job should go in, etc. Then it takes this script, cmd.sh and executes it in a system specific way, e.g. by doing: os.system("qsub cmd.sh") – Adjutant

It seems the problem stems from the fact you insist on running Python script from cmd.sh shell script. If I understand above you transfer cmd.sh script to the target machine, where the task will be run. Can't you transfer script.py the same way so that you know the location of this script on the target machine? – Slavic

Answer 1

I think you should structure your code so that you don't need to call scripts from you code. Move code you need from scripts to your package and then you can call this code both from your scripts and from your code.

Slavic

I like the idea but in practice it does not work for me see upcoming edit – Adjutant

I see nothing in your workflow preventing moving some code from script to the package. – Slavic

I need to be able to make a command that calls a Python script, which is part of my package. Something like "python myscript.py someargs". Therefore, I need to know where myscript.py lives... Even myscript.py was part of the package and not in the scripts dir, I'd still need to use a hack like __file__ to figure out where the script is. Would that be preferable from a design perspective rather than trying to find out where the scripts directory is? Both solutions seem inelegant – Adjutant

Answer 2

My use case for this was to check that the directory my scripts are installed into is in the user's path and give a warning if not (since it is often not in the path if installing with --user). Here is the solution I came up with:

from setuptools.command.easy_install import easy_install

class my_easy_install( easy_install ):

    # Match the call signature of the easy_install version.
    def write_script(self, script_name, contents, mode="t", *ignored):

        # Run the normal version
        easy_install.write_script(self, script_name, contents, mode, *ignored)

        # Save the script install directory in the distribution object.
        # This is the same thing that is returned by the setup function.
        self.distribution.script_install_dir = self.script_dir

...

dist = setup(...,
             cmdclass = {'build_ext': my_builder,  # I also have one of these.
                         'easy_install': my_easy_install,
                        },
            )

if dist.script_install_dir not in os.environ['PATH'].split(':'):
    # Give a sensible warning message...

I should point out that this is for setuptools. If you use distutils, then the solution is similar, but slightly different:

from distutils.command.install_scripts import install_scripts

class my_install_scripts( install_scripts ):  # For distutils
    def run(self):
        install_scripts.run(self)
        self.distribution.script_install_dir = self.install_dir

dist = setup(...,
             cmdclass = {'build_ext': my_builder,
                         'install_scripts': my_install_scripts,
                        },
            )

Cruiser

Answer 3

I think the correct solution is

scripts=glob("myscripts/*.py"),

Revanche