PyInstaller --hidden-import wildcard?

Question

I'm trying to make executable from Scrapy project. I've noticed that I have to say PyInstaller what scrapy modules it has to load. The problem is that there is a lot of these modules.

pyinstaller --onefile main.py --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.....

Is it possible to set PyInstaller to preimport all modules? Something like --hidden-import scrapy.* which doesn't work.

Bandylegged

Answer 1

Really late to the discussion, but just encountered the same problem and this is the sollution I came to in the end.

1. Use .spec files for generating your executable. .spec files are actually interpreted as python files and you can include your own code for setting up PyInstaller settings.

2. Make a function that lists all of your hiddenimports. For my use case, I used dynamic imports from folders in the project, but you can use the same idea and other code to list all modules you need. This is how my .spec file looks like:

# -*- mode: python ; coding: utf-8 -*-


block_cipher = None

# Generate list of hidden imports
import os
def list_python_files(dir_name):
    file_list = []
    for filename in os.listdir(dir_name):
        if filename.endswith('.py'):
            file_list.append(f"{dir_name}.{filename[:-3]}")
    return file_list

hiddenimports = list_python_files('folder_name_here')

a = Analysis(
    ['app.py'],
    pathex=[],
    binaries=[],
    datas=[('icon.ico', '.')],
    hiddenimports=hiddenimports,
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False,
)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)

exe = EXE(
    pyz,
    a.scripts,
    [('v', None, 'OPTION')],
    exclude_binaries=False,  # changed to False
    name='app',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    console=False,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
    icon='icon.ico',
)
coll = COLLECT(
    exe,
    a.binaries,
    a.zipfiles,
    a.datas,
    strip=False,
    upx=True,
    upx_exclude=[],
    name='app',
)

Redress

The question is tagged python-2.7 but new code should probably use pathlib instead of os and os.path functions to deal with path and file names. Also this example only works with a single non nested package in the current working directory where each module is a *.py file. System wide installed nested packages maybe with modules implemented in C will need some additional code. – Orangery

Answer 2

The 2023 solution is detailed here: https://github.com/pyinstaller/pyinstaller/issues/1905#issuecomment-525221546

Basically you need to create an appropriate hook.

Ensure that a hook injects the "hidden" modules from your plugins folder inside the executable:

• if your plugins are under the myappname/pluginfolder module
• create a file specs/hook-<myappname.pluginfolder>.py
• content of this file should be:

from PyInstaller.utils.hooks import collect_submodules
hiddenimports = collect_submodules('<myappname.pluginfolder>')

Picklock

Answer 3

Pyinstaller should have created a "main.spec" file. In that file, there is a line containing "hiddenimports=[]". That is where you can list all the hidden imports and only do it once. You may be able to use wildcards in that list, but I'm not sure of that.

Heall