Pyinstaller - libmagic not found after compiling app

Question

Problem

I'm writing an application in Python. It works just fine in my python environment, but when I compile it with PyInstaller and try to run the resulting executable, I get the following error: ImportError: failed to find libmagic. Check your installation

I have a feeling that this has something to do with "python-magic-bin", an install containing the binaries that were required for a module named "magic", because if I uninstall "python-magic-bin" from my Python environment and try to run the app, I get the same exact error. I think that for some reason these binaries aren't being carried to the compiled exe.

If it makes a difference, I installed the "python-magic-bin" from a .whl file. This install added a folder called "libmagic" files libmagic.dll and magic.mgc to the directory that "magic" was installed to.

Question

If I'm right about the problem, how do I get PyInstaller to carry over the binaries that "magic" needs?

Recreate the Problem

The following can be done to recreate the problem:

1. Copy and paste this code into your editor. Save it as a file called "test.py"

   import magic
   
   m=magic.MAGIC_NONE
   print(m)
   

2. Download "python_magic_bin-0.4.14-py2.py3-none-win32.whl" from this link and use the following commands in the interpreter to install the required libraries to Python 3.6

   >>> pip install pyinstaller
   >>> pip install python-magic
   >>> pip install python_magic_bin-0.4.14-py2.py3-none-win32.whl
   >>> pip install libmagic
   

3. Open a command prompt in the same directory as that "test.py" file and use the following command to compile the program using pyinstaller:

   > pyinstaller test.py
   

4. After it's done, move to the newly created /dist/test directory (cd ./dist/test) and run the .exe using:

   > ./test.exe
   

After running it, you should see an error reading: ImportError: failed to find libmagic. Check your installation and Failed to execute script test

Spec File

This is the spec file I'm using to compile my project.

# -*- mode: python -*-

block_cipher = None


a = Analysis(['main.py'],
             pathex=['D:\\Home_Python\\pytags'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)

pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)

exe = EXE(pyz,
          a.scripts,
          exclude_binaries=True,
          name='main',
          debug=False,
          strip=False,
          upx=True,
          console=True )

coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               name='main')

Versions

For this project, I'm using:

• Python 3.6.3
• PyInstaller 3.3.1
• Libmagic 1.0
• Python-Magic 0.4.15
• Python-Magic-Bin 0.4.14

Manthei

You will get more and better answers if you create a Minimal, Complete, and Verifiable example. Especially make sure that the input and expected data are complete (not pseudo-data), and can be easily cut and and paste into an editor to allow testing proposed solutions. – Membrane

I actually liked how the OP phrased the question - I'm struggling with the same thing. I suspect that pyinstaller needs something in C:\Python27\Lib\site-packages\PyInstaller\hooks that it doesn't have. – Mellophone

Suzanne, I actually figured out the problem not too long ago. I can't post it right now, but I'll reply to this thread with the solution when I can. – Manthei

I had a similat problem with a package that I installed in a virtual environment only but had pyinstaller pull from the root pythn installation on C:\python. Did not give an error during pyInstaller compile but at runtime. Installed the package with pip und it worked afterwards – Pantalets

Answer 1

I figured the problem out after a couple of days, so I'm going to document my solution here for anyone who may be having the same problem in the future.

The issue is the method that libmagic uses to find the .dll file it needs to work properly.

Head over to Lib/site-packages/magic under your Python installation and copy the file named magic.py and the folder named libmagic into the directory of your Python project. After doing that, open up magic.py in your preferred IDE. If you head over to line 156, you'll see this bit of code:

bin_dist_path = os.path.join(os.path.dirname(__file__), 'libmagic')

This is the line that is causing our problem. It looks for a folder called libmagic in the same directory as the magic.py file. Specifically, the issue here is that the __file__ variable does not work the same when the file is frozen into a .exe file so I replaced __file__ with sys.executable. The resulting line looked like this:

bin_dist_path = os.path.join(os.path.dirname(sys.executable), 'libmagic')

Save the file, compile your program, and copy the libmagic folder into the same directory as the resulting .exe file.

If you run the .exe, everything should now work properly.

If there's anything I could clarify or you have questions, please don't hesitate to ask.

Manthei

I will note that I also copied libmagic.dll and magic.mgc in the binaries= section of my .spec file. – Reahard

If you don't want to manually copy the libmagic folder you can add in the spec file: extra_files += [('venv/Lib/site-packages/magic/libmagic','libmagic')] and use that for "datas". – Tadtada

Answer 2

Didn't want to put this in another comment because it wouldn't be so visible...

This works for me. Just change the .spec file to include extra files:

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None

extra_files = [('venv/Lib/site-packages/magic','magic')]

a = Analysis(['main.py'],
             pathex=['D:\\Home_Python\\pytags'],
             binaries=[],
             datas=extra_files,
             hiddenimports=[],

...

Tadtada