Installing python packages locally doesn't always work

Question

I'm creating a python 3.9 program and want to install packages locally. So the way my project is set up is this:

__main__.py
test.py
requirements.txt
lib/
    __init__.py

In my requirements.txt file I have 3 lines:

colorama==0.2.2
click==8.0.3
pendulum==2.1.2

Then I run: python -m pip install -r requirements.txt -t ./lib This installs all the packages and dependencies inside of the lib directory.

Then I import the modules at the top of my test.py file:

from lib import colorama
from lib import click
from lib import pendulum

In doing some testing, I've found that colorama works fine. I'll use it in a simple test: print(colorama.Fore.BLUE + "Hello, World!"). The text is blue in the console and everything is working.

I then try to use the other packages and I get ModuleNotFoundError exception: print(pendulum.now('Europe/Paris'))

Exception has occurred: ModuleNotFoundError - No module named 'pendulum' This is coming from one of its own files.

The same thing happens when I use Click, but it's a little different. I'll get the same ModuleNotFound exception, but it's for its own dependency on Colorama. I don't think it's related to the fact that I'm also importing Colorama because if I uninstall I get the same error.

I've also tried this with the python-docx package. I added python-docx==0.8.11 to the requirements.txt file, then issued the same command as above to install to my local lib directory. It seems to install fine. I see the docx directory and all its dependencies. Then I import from lib import docx then do something simple in test.py: doc = docx.Document()

Then get ModuleNotFound error: File "C:\Users\name\Development\python\test-local-package\lib\docx_init_.py", line 3, in (Current frame) No Module named 'docx'

Does anyone know what I'm doing wrong?

Thousandth

Try adding the full path to the "lib" folder to the module search path. via sys.path.append(<full path to lib>) – Damron

Where do I add that? I'm a python noob – Thousandth

Plus, I want to be able to package this up in a zip file when I'm done, so it should be able to work with the current working directory and not a full path – Thousandth

import sys as line 1 and then sys.path.append(<full path to lib>) as line 2 then the remainder of your imports – Damron

The interesting thing is that my IDE shows that the modules exist when I import them from lib – Thousandth

Answer 1

When you put those libraries into your lib folder and import them the way you are doing, you're changing their package names. No longer is colorama a top-level package, it's now lib.colorama. Some libraries might be fine with that, but for others, they expect to be able to import their own code using their normal names. If colorama.some_submodule tries to import colorama, it will fail.

It's important to realize that a statement like from lib import colorama doesn't change how colorama can be found everywhere. It only changes the local namespace. The package is still lib.colorama, we've just bound it to the name colorama in the current module.

As JonSG has suggested in comments, a better solution is to put the lib folder into the Python search path so that import colorama will find the package with its normal name. Modifying sys.path is one way to do that, another is the PYTHONPATH environment variable (probably not ideal for your current issue, but sometimes useful in other situations).

Thebes

That's interesting. I was under the impression that putting the empty init.py file in the lib directory would resolves those issues. What I want to do (in the end) is zip the whole directory from the root, then use the zip file like: 'python my_prog.zip' to run the program. So how can I do that so that the packages in lib will work without having to rely on a system path? – Thousandth