How to build pex or shiv package from pyproject-compliant project?

Question

I have a Python project which I would like to distribute as a Pex or shiv self-contained Python-executable package, in the spirit of the Python Packaging Guide, "Depending on a pre-installed Python" section. My project is structured in the spirit of PEP518, and it has a pyproject.toml file. My project also includes a few libraries not in the Python Standard Library, so I use pipenv to manage those.

How to I build the pex package using a backend which I can specify in the [build-backend] of my pyproject.toml file?

The documentation for pex and shiv show how to build self-contained packages from the command line, or via setuptools.py, but not using the PEP518 structure and pyproject.toml. At least, not as far as I have been able to discover. (And, by "self-contained", I mean all Python language packages, but I am happy to use an existing Python 3 interpreter on the destination system.)

Note that of the three executable packages listed in the Packaging Guide, zipapps does not seem like a fit for me. It doesn't give me a way to manage my external libraries.

Update: some specific invocations, per request.

I currently use build as my build frontend. I use setuptools as my build backend. My pyproject.toml file currently reads,

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

I currently build a wheel via this shell command:

(MyPipenvVenv) % python -m build
…[many lines of output elided]…
Successfully built MyProject-0.0.6a0.tar.gz and MyProject-0.0.6a0-py3-none-any.whl

I can build a self-contained app (which relies on the system's Python interpreter) using these pipenv and shiv commands:

(MyPipenvVenv) % pipenv requirements > requirements.txt
(MyPipenvVenv) % shiv --console-script myapp -o app/myappfile.pyz -r requirements.txt .
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting click==8.1.3
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting pip==22.1.2
  Using cached pip-22.1.2-py3-none-any.whl (2.1 MB)
Collecting setuptools==62.5.0
  Using cached setuptools-62.5.0-py3-none-any.whl (1.2 MB)
Collecting shiv==1.0.1
  Downloading shiv-1.0.1-py2.py3-none-any.whl (19 kB)
Building wheels for collected packages: MyProject
  Building wheel for MyProject (pyproject.toml): started
  Building wheel for MyProject (pyproject.toml): finished with status 'done'
  Created wheel for MyProject: filename=MyProject-0.0.6a0-py3-none-any.whl size=5317 sha256=bbcc…cf
  Stored in directory: /private/var/folders/…/pip-ephem-wheel-cache-eak1xqjp/wheels/…cc1d
Successfully built MyProject
Installing collected packages: MyProject, setuptools, pip, click, shiv
Successfully installed MyProject-0.0.6a0 click-8.1.3 pip-22.1.2 setuptools-62.5.0 shiv-1.0.1

What I want is to give the command to the PEP 517 front-end, have the pyproject.toml specify that the resulting build work be done by shiv, and point to whatever configuration shiv needs. I want the result be a self-contained app file app/myappfile.pyz. e.g.

(MyPipenvVenv) % python -m build
…[many lines of output elided]…
Successfully built MyProject
Installing collected packages: MyProject, setuptools, pip, click, shiv
Successfully installed MyProject-0.0.6a0 click-8.1.3 pip-22.1.2 setuptools-62.5.0 shiv-1.0.1

My pyproject.toml file would be something like,

[build-system]
requires = ["shiv"]
build-backend = "shiv.build_something_something"

Skyler

What have you concretely tried? What did not work as expected, and what were you expecting? -- What I understood from what you described should work, seems straightforward to me. – Singletree

@Singletree The documentation for pex and shiv show how to build self-contained packages from the command line, or via setuptools.py, but not using the PEP518 structure and pyproject.toml. In particular, it is not clear to me what value to put for the build-backend key in pyproject.toml to invoke pex or shiv. – Skyler

Nothing needs to be changed in pyproject.toml. -- You need to call the pex or shiv CLI tool directly and you get a redistributable zipapp / *.pyz file as output. From memory, it is as easy as giving the path to the project directory as argument to the pex or shiv CLI tool (the directory containing the pyproject.toml). Pex and Shiv should be able to figure out the build back-end and take it from there. – Singletree

@sinoroc, I think your comments amount to an answer: the pex and shiv tools do not support the spirit of PEP518, one cannot build a pex package using a backend specified in the build-backend of the pyproject.toml file, instead one must disregard PEP518 and invoke the backend directly. I'll be disappointed if that is the answer, but maybe it is. – Skyler

No, that is not what I meant. I feel like you are worrying about the wrong things here. Pex and Shiv do very much take into account the build back-ends. -- Go to the directory containing pyproject.toml, run something like pex . -o app.pyz (check doc for exact command) then run ./pex.pyz. -- Really, just actually try using pex and shiv, they will use the correct build back-ends, no worries. – Singletree

Pex and shiv are more on the front-end side of the build. I suspect they both actually delegate to pip (or an other actual build front-end) whenever something needs to get built. And in many cases nothing even needs to get built at all, since pre-built wheel files can be used directly. -- So nothing needs to be modified in the build back-end config, and that is exactly the point of PEP 518, a clear interface between front-end and back-end. Whatever the front-end is being used, the back-end can stay the same. – Singletree

I agree with you on the purpose of PEP518, a clear separation between front-end and back-end. But I have a different understanding about where ''pex'' and ''shiv'' fit in that model. It seems like they do back-end work, building the executable file that gets deployed. PEP 517 is not terribly clear in its definitions of "build front-end" and "build back-end" though: peps.python.org/pep-0517/#terminology-and-goals – Skyler

Pex and shiv are definitely on the front-end side of things. Pex and shiv definitely do not do back-end work. – Singletree

This discussion does not seem to be going anywhere. I recommend you either rephrase your original post entirely to ask a single question with examples of things that you have tried (minimal reproducible example), or post a new question entirely, one that fits the StackOverflow guidelines. -- On the other hand, if you need clarification on terminology, on PEP 517, PEP 518, pex, shiv, Python packaging in general and discuss more thoroughly, feel free to post here. – Singletree

Answer 1

As far as I know, shiv is not a "PEP 517 build back-end" (neither is pex), so it is not possible to write something like the following in pyproject.toml:

[build-system]
requires = ["shiv"]
build-backend = "shiv.build_something_something"

As discussed there, the PEP 517 interface is targeted at the generation of source distributions (sdist) and wheels only.

From my point of view, I consider tools like shiv and pex that generate zipapps to be (at least) one layer above. And when working at this level, it does not matter whether or not sdists and/or wheels are generated via the PEP 517 interface, in other words it does not matter whether or not pyproject.toml files are involved. I assume that shiv and pex either consume wheels and sdists that are already available (maybe downloaded from PyPI) or they delegate the "build" step to a 3rd party tool (maybe pip, maybe build), I do not know and it does not matter.

From my point of view, the input that makes the most sense to get a zipapp as output is some kind of "lock file", and not a (PEP 517) pyproject.toml file. Zipapps are basically one whole "virtual environment" in a single file. It means that the Python interpreter is fixed, and each dependency (direct or indirect) is fixed. This is best described with a lock file.

The requirements.txt files while not strictly lock files, are probably what is the closest thing with enough availability and support in the Python packaging ecosystem. And as far as I know the requirements.txt files are the only "lock file"-ish format that tools like shiv and pex accept as input.

So my recommendation for you would be to focus on requirements.txt files to provide as input to pex or shiv. As you are already doing.

---

In the Python packaging ecosystem...

It looks like PDM has a real lock file format and already has support for generating zipapps via a plugin pdm-packer.

Poetry also has a lock-file format and they are somewhat looking into supporting zipapps as well

There are discussions and work going on towards a standardized lock file format. But it is difficult work, and will probably still take some time to reach a conclusion.

Singletree