Pyinstaller on a setuptools package

Question

I'm attempting to run PyInstaller on a CLI app I am building in Python using the Click library. I'm having trouble building the project using PyInstaller. PyInstaller has a document in their GitHub wiki titled Recipe Setuptools Entry Point, which gives information about how to use PyInstaller with a setuptools package, which I'm using for this project. However, it seems it cannot find the base module when I run pyinstaller --onefile main.spec.

My question is: Is the problem simply an issue with the folder structure I have? Does the Recipe Setuptools Entry Point assume a certain file structure?

Relevant information

Pyinstaller output

184 INFO: PyInstaller: 3.3.1
184 INFO: Python: 3.6.4
189 INFO: Platform: Darwin-16.7.0-x86_64-i386-64bit
193 INFO: UPX is available.
Traceback (most recent call last):
  File "/usr/local/bin/pyinstaller", line 11, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.6/site-packages/PyInstaller/__main__.py", line 94, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/usr/local/lib/python3.6/site-packages/PyInstaller/__main__.py", line 46, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 791, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/usr/local/lib/python3.6/site-packages/PyInstaller/building/build_main.py", line 737, in build
    exec(text, spec_namespace)
  File "<string>", line 40, in <module>
  File "<string>", line 26, in Entrypoint
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 582, in get_entry_info
    return get_distribution(dist).get_entry_info(group, name)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 564, in get_distribution
    dist = get_provider(dist)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 436, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 984, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 870, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'myapp' distribution was not found and is required by the application

The main.spec file for main.py, which is the entrypoint for my CLI app:

block_cipher = None

def Entrypoint(dist, group, name,
               scripts=None, pathex=None, hiddenimports=None,
               hookspath=None, excludes=None, runtime_hooks=None):
    import pkg_resources

    # get toplevel packages of distribution from metadata
    def get_toplevel(dist):
        distribution = pkg_resources.get_distribution(dist)
        if distribution.has_metadata('top_level.txt'):
            return list(distribution.get_metadata('top_level.txt').split())
        else:
            return []

    hiddenimports = hiddenimports or []
    packages = []
    for distribution in hiddenimports:
        packages += get_toplevel(distribution)

    scripts = scripts or []
    pathex = pathex or []
    # get the entry point
    ep = pkg_resources.get_entry_info(dist, group, name)
    # insert path of the egg at the verify front of the search path
    pathex = [ep.dist.location] + pathex
    # script name must not be a valid module name to avoid name clashes on import
    script_path = os.path.join(workpath, name + '-script.py')
    print ("creating script for entry point", dist, group, name)
    with open(script_path, 'w') as fh:
        print("import", ep.module_name, file=fh)
        print("%s.%s()" % (ep.module_name, '.'.join(ep.attrs)), file=fh)
        for package in packages:
            print ("import", package, file=fh)

    return Analysis([script_path] + scripts, pathex, hiddenimports, hookspath, excludes, runtime_hooks)

a = Entrypoint('myapp', 'console_scripts', 'myapp')

pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='main',
          debug=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               name='main')

The contents of the myapp script generated when I run pip3 install --editable . in my virtual environment:

#!/some/path/to/myapp-cli/venv/bin/python3.6
# EASY-INSTALL-ENTRY-SCRIPT: 'myapp','console_scripts','myapp'
__requires__ = 'myapp'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('myapp', 'console_scripts', 'myapp')()
    )

And finally, my repository structure:

myapp-cli/
├── README.md
├── myapp
│   ├── __init__.py
│   ├── main.py
│   ├── main.spec
│   ├── resources
│   │   ├── __init__.py
│   │   └── functions.py
│   ├── subcommands
│   │   ├── __init__.py
│   │   ├── config
│   │   │   ├── __init__.py
│   │   │   └── cli.py
│   │   ├── create
│   │   │   ├── __init__.py
│   │   │   └── cli.py
│   │   ├── destroy
│   │   │   ├── __init__.py
│   │   │   └── cli.py
│   │   └── switch
│   │       ├── __init__.py
│   │       └── cli.py
│   └── variables.py
├── requirements.txt
└── setup.py

And my setup.py file:

from setuptools import find_packages
from setuptools import setup
import os

base_dir = os.path.dirname(__file__)

setup(
    entry_points = '''
        [console_scripts]
        myapp=myapp.main:entry_point
    ''',
    install_requires = [
        'packageone==1.0',
        'packagetwo==2.0',
    ],
    name = "myapp",
    packages=find_packages(),
    setup_requires="setuptools",
    version = "0.1",
)

Greet

Answer 1

First: I used a combination of Stephen's answer, and some digging of my own to find the answer. In the end, Stephen's first part did the trick: manually adding / exporting the PYTHONPATH variable. You can actually specify this using pathex in the Entrypoint function like so:

a = Entrypoint('myapp-cli',
    'console_scripts',
    'myapp',
    pathex=['/some/path/to/myapp-cli/myapp', '/some/path/to/myapp-cli']
)

I didn't end up needing the myapp.main after all.

Second: I was still having issues with PyInstaller not producing a single binary. For me, this did the trick:

• Add the latest version of PyInstaller to your requirements.txt or to your install_requires in setup.py: https://github.com/pyinstaller/pyinstaller/archive/develop.zip.
• Also, you can make your .spec file with the --onefile option in pyi-makespec like so: pyi-makespec --onefile myapp.py. This will make a .spec file that ensures that all of your packages are compiled into the binary.

In the end, the following spec file did the trick, and I was able to make a fully working binary:

# -*- mode: python -*-

block_cipher = None

def Entrypoint(dist, group, name,
               scripts=None, pathex=None, hiddenimports=None,
               hookspath=None, excludes=None, runtime_hooks=None):
    import pkg_resources

    # get toplevel packages of distribution from metadata
    def get_toplevel(dist):
        distribution = pkg_resources.get_distribution(dist)
        if distribution.has_metadata('top_level.txt'):
            return list(distribution.get_metadata('top_level.txt').split())
        else:
            return []

    hiddenimports = hiddenimports or []
    packages = []
    for distribution in hiddenimports:
        packages += get_toplevel(distribution)

    scripts = scripts or []
    pathex = pathex or []
    # get the entry point
    ep = pkg_resources.get_entry_info(dist, group, name)
    # insert path of the egg at the verify front of the search path
    pathex = [ep.dist.location] + pathex
    # script name must not be a valid module name to avoid name clashes on import
    script_path = os.path.join(workpath, name + '-script.py')
    print ("creating script for entry point", dist, group, name)
    with open(script_path, 'w') as fh:
        print("import", ep.module_name, file=fh)
        print("%s.%s()" % (ep.module_name, '.'.join(ep.attrs)), file=fh)
        for package in packages:
            print ("import", package, file=fh)

    return Analysis([script_path] + scripts, pathex, hiddenimports, hookspath, excludes, runtime_hooks)

a = Entrypoint('myapp-cli',
    'console_scripts',
    'myapp',
    pathex=['/some/path/to/myapp-cli/myapp', '/some/path/to/myapp-cli']
)

pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          name='myapp',
          debug=False,
          strip=False,
          upx=True,
          runtime_tmpdir=None,
          console=True )

I think in the end using something like Cobra for Golang would work easier since Golang compiles one-file binaries out of the box. However, if you prefer Python, this should do the trick.

Greet

Answer 2

This error:

pkg_resources.DistributionNotFound: The 'myapp' distribution was not found and is required by the application

indicates that this package is not on PYTHONPATH. I fixed it on Windows with:

set PYTHONPATH=.

adjust to your OS of choice.

---

In addition to the path problem, there is:

In setup.py:

setup(
    entry_points = '''
        [console_scripts]
        myapp=myapp.main:entry_point
    ''',

In main.spec:

a = Entrypoint('myapp', 'console_scripts', 'myapp')

According to setup.py, it looks like your entry point is myapp.main not myapp. So you may need:

a = Entrypoint('myapp', 'console_scripts', 'myapp.main')

Waver

Answer 3

The accepted answer didn't work for me. I had to add the egg-info directory via the .spec file.

My call to the Entrypoint function looks like this:

a = Entrypoint(
        'PrintIt',
        'console_scripts',
        'printit',
        datas=[('plugins/*.egg', 'plugins/'),
               ('../PrintIt.egg-info/*', 'PrintIt.egg-info/')])

Stedmann

Answer 4

Something I've noticed is that the typical way of adding a data file doesn't work once you've monkey patched Entrypoint in the way Scott Crooks recommends in the ticked answer. For me, I had to append to the a.datas array. In python3, this looks like:

...
a = Entrypoint(...)
from pathlib import Path
Path('/tmp/modulename/datafile.txt').write_text(Path('datafile.txt').read_text()))
a.datas.append('datafile.txt', '/tmp/modulename/datafile.txt', 'DATA')

pyz = PYZ(...)
...

Sn

Answer 5

After much searching, this error is generally due to attempting to access your project's package's metadata (i.e. version being the primary one).

Package metadata is typically accessed using pkg_resources or the older distutil, either explicitly, or often hidden in other packages (usually attempting to access the package version). Starting with Python v3.8 it will also be available in the stdlib within importlib.metadata.

If this is the case, you likely need to include some or all of the files in a mypackage.egg-info folder, especially the file PKG_INFO, but it may require them all.

There are multiple ways to do this, here are several I like:

---

1. If you are using a script.spec file, you can update the datas= line to include this info, per Charles answer:

a = Analysis(['myscript.py'],
             pathex=['C:\\path\\to\\mypackage'],
             binaries=[],
             datas=[('mypackage.egg-info/*','mypackage.egg-info')],

---

2. Create a custom hook file, put it in a directory, and add the directory as a custom hooks directory at the command line.

Create a hook-mypackage.py hook file, with the following very simple, and pretty elegant, lines:

from PyInstaller.utils.hooks import copy_metadata

datas = copy_metadata('md2mat')

I put this into a new hooks folder in my root package/repo folder, then added the following to my pyinstaller command:

pyinstaller -F -y --additional-hooks-dir=hooks myscript.py

It works pretty well, and assuming the copy_metadata function is well maintained as we change over from older metadata packages to the new importlib.metadata, it should work well through future Python updates.

---

3. Add additional data files directly at the command-line

This might be my favorite, if I could get it working...

pyinstaller --add-data <SRC;DEST> myscript.py

This option --add-data shows up in the help output (pyinstaller --help), and indicates the format of the argument should be SRC;DEST for Windows, so I think it must match the datas= format from the other methods, but I couldn't get it to work.

The closest I think I got to the right format were the following:

pyinstaller -F -y --add-data "mypackage.egg-info/*;mypackage.egg-info"
pyinstaller -F -y --add-data="mypackage.egg-info/*;mypackage.egg-info"

These would compile, but the resulting exe would just run with no output.

The --add-data option is missing from the PyInstaller Documentation, but shows up when running pyinstaller --help-commands.

Latisha