Python PEX loading

Question

I've been trying to wrap my head around the python pex utility (https://pex.readthedocs.org/en/latest/) for bundling some applications into .pex files for deployment.

My app isn't large enough to require twitters pants build tool, as well as I have some build requirements which pants doesn't address. I did however, try the pants tools build system using python_binary which resulted in pex files with the sources loaded into the pex files root. The BUILD files in pants accept a sources property for python_binary which can be a glob for files within the directory the build is running in, however, pants is using the pex programmatic API and not the command-line utility.

The problem is when I use the pex command-line utility on it's own, it seems to want distributions (i.e. folders set up with a setup.py, etc...) and wants to install my code into the .deps folder in the pex file rather than just copying the python files into the root of the pex file like pants seems to do.

Is copying the files over (not installing a package) not possible through the command-line pex tool?

Midget

Answer 1

As of pex 1.0.0 there is no facility to glob up files and directories directly, you must have a setup.py as you suggest, or use pants (no longer Twitter's by the way - independent).

So you have 3 paths forward (#1 you already know, but spelling it out for others):

1. Create a setup.py and point the pex tool at its dir

   $ tree -h
   .
   ├── [4.0K]  lib
   │   ├── [   0]  __init__.py
   │   ├── [  38]  lib.py
   │   └── [  68]  main.py
   └── [  76]  setup.py
   
   1 directory, 4 files
   $ cat lib/lib.py 
   def func():
       return 'func in lib'
   
   $ cat lib/main.py 
   from .lib import func
   
   if __name__ == '__main__':
       print(func())
   
   $ cat setup.py 
   from setuptools import setup
   
   setup(
       name='lib',
       packages=['lib']
   )
   
   $ pex . -e lib.main -o lib.pex
   $ ./lib.pex 
   func in lib
   

   NB: The . in the pex command line is the bit pointing pex at this dir's setup.py

2. File an issue against pex to support a set of files in place of a requirement / setup.py. You can do that here.

3. File issues against pants to support the requirements you have that it does not address. You can do that here

As a pants committer I can say that we're working towards making pants easier and easier to use such that no project is too small. You should be able to pip install pantsbuild.pants.backend.python && touch pants.ini and be up and running in a python-only repo but we're not there today.

Rouault

Marking this as the answer, thanks for laying it out plainly, Will look at pex and create an issue to see if they'd merge a new option to support globs if I worked on it. The reason I won't do pants (and i'm not saying it's a bad tool, it's actually really awesome) is that I have builds of various components in a monolithic repo that pants doesn't do. The repo has python items which pants works great for, but it also has about 30 JS React-based apps that need to be built as well as config merges, SQL install/updates and I couldn't easily figure out how to support those in pants. – Midget