python does not find module installed with pipx [duplicate]

Question

Debain stable wants me to install Python modules using pipx. So I do

$ pipx install auditwheel
$ pipx ensurepath
$ python3 -m pipx ensurepath
$ python3
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import auditwheel
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'auditwheel'
>>> 

What am I doing wrong?

Nara

Thats the feature of pipx - if you install auditwheel and ensurepath with pipx - both of those are installed into separate "virtual environments" and they are available only if the venv of those is activated. Which happens when their respected cli entrypoints are executed. TL;DR pipx is intended to be used to install standalone applications that are completely isolated from any other python env. – Ripsaw

aka, use pip instead of pipx and check out how to utilize virtual environments. – Ripsaw

Please read Debian's advice more carefully. pipx is for installing applications, not libraries. – Disingenuous

Answer 1

From Python 3.11 onward, Debian encourages the users to create a separate Python virtual environment to install Python packages.

Because Debian declares its Python install to be externally-managed, pip (and other installers) will refuse to install packages system-wide. Installation is only possible in virtual environments or separate Python installs. This is because Python package installers (like pip) are unaware of the constraints that APT-managed packages have on libraries and versions. See PEP-668 for a full discussion of the problems that can occur when multiple installers operate on the same Python install.

Therefore, the optimal way is to create a virtual environment, say MyEnv, and install packages therein:

$ mkdir -p $HOME/.venvs  # create a folder for all virtual environments 
$ python3 -m venv $HOME/.venvs/MyEnv  # create MyEnv

This will create a directory $HOME/.venvs/MyEnv with a configuration file pyvenv.cfg which includes some details for this virtual environment, such as the Python executable and Python version.

Verify the version of the Python in the virtual environment:

$HOME/.venvs/MyEnv/bin/python --version

The executables of the created virtual environment are found under $HOME/.venvs/MyEnv/bin.

To install a package into the virtual environment, use

$HOME/.venvs/MyEnv/bin/python -m pip install <some-package>

To 'activate' the virtual enviroment, i.e. adding its configuration variables into the shell environment, use

source $HOME/.venvs/MyEnv/bin/activate

Consult Python's guide to virtualenv and pip at https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments.

Neoarsphenamine

how does this refers to the actual question ? Of course, pipx uses virtual environments but it does not address "what did i do wrong" .. – Ripsaw

but kudos for preaching about venv's – Ripsaw

@Ripsaw It refers to the actual question because it explains the correct way to solve the problem that OP incorrectly tried to solve by using pipx. It addresses "what did I do wrong", because what OP did wrong is trying to use pipx for this problem. pipx manages virtual environments, but in a way that is not suitable for OP's purposes. – Disingenuous

Answer 2

pipx is not exactly the same as pip. pipx installs applications in an isolated environment. It will not help if you want to compile and import a module. You can use a virtual environment as @AlQuemist suggests.

Kooima

This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review – Epimorphosis

@Epimorphosis It does provide an answer to the question. It's just expanding on another answer. – Julee

It absolutely does answer. Q. "What am I doing wrong?" A. "Using pipx; it isn't the right tool for your purposes." – Disingenuous

Answer 3

Message in pip3 might be misleading when it comes to pipx as it is there mainly to get binaries produced by python packages into ~/.local/bin and not to system-wide location.

You can still use pipx for single package, it creates virtual environment for each package installed. You can run e.g.

~/.local/pipx/venvs/PACKAGENAME/bin/python3

If you are OK with doing what Debian developers don't want and the level of isolation to install additional packages to user dir is sufficient for you, the easiest is to

pip3 install --break-system-packages --user package

To clarify use cases

pipx

You want yt-dlp package and you are not exactly interested in importing it when programming youself in python but you are only interested in the binary it builds. So you do pipx install yt-dlp and you can use yt-dlp binary.

pip, pip3

Your python code is dependent on the package bitcash not available as a system package. You then run pip install bitcash # pip3 if pip is Python 2 on your system``.

Manpage say: On Debian, pip is the command to use when installing packages for Python 2, while pip3 is the command to use when installing packages for Python 3. It has some "clever functions" s.t. pip provides pip3 functionality when there is no python2 installed. That might be bit confusing.

Travesty

Thank you. Can you possibly explain in more detail pip vs pip3 vs pipx? I find this extremely confusing. – Nara

Answer 4

Debian docs at https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.html#python3-pep-668

If you need to install a Python application (or version) that isn't packaged in Debian, we recommend that you install it with pipx (in the pipx Debian package). pipx will set up an environment isolated from other applications and system Python modules, and install the application and its dependencies into that.

Emphasis mine, the docs do talk about application and under that paragraph is second one pointing to installing packages to the virtual environment;

If you need to install a Python library module (or version) that isn't packaged in Debian, we recommend installing it into a virtualenv, where possible. You can create virtualenvs with the venv Python stdlib module (in the python3-venv Debian package) or the virtualenv Python 3rd-party tool (in the virtualenv Debian package). For example, instead of running pip install --user foo, run: mkdir -p ~/.venvs && python3 -m venv ~/.venvs/foo && ~/.venvs/foo/bin/python -m pip install foo to install it in a dedicated virtualenv.

For which @AlQuemist's answer also highlights ..

Ripsaw

I believe using Python's universal venv explicitly is far better than arbitrary tools like pipx, esp. for developers living on different platforms. – Neoarsphenamine

Not disagreeing with that ;) its all just PATH shuffling – Ripsaw