ModuleNotFoundError: No module named '_lzma' when building python using pyenv on macos

Question

Trying to use pyenv to build python versions on macOS, but every install I make ends up with

❯ python
>>> import lzma

ModuleNotFoundError: No module named '_lzma'

I have installed all of the following packages:

brew install openssl readline sqlite3 xz zlib
xcode-select --install

Only caveat is that homebrew installs packages to ~/.brew.

Any input is appreciated.

Handout

Do you get the same error if you run the code from a file instead of the REPL? – Komsomol

@Komsomol I do receive the same error if I run python my_script.py – Handout

This exact error seems to be relatively popular: github.com/pandas-dev/pandas/issues/27532, github.com/pandas-dev/pandas/issues/27543, https://mcmap.net/q/225937/-modulenotfounderror-no-module-named-39-_lzma-39/11301900 – Komsomol

Yeah, thanks! The typical solution seems to be "install the correct packages before building". Unfortunately, to the best of my knowledge, I have already installed all the relevant packages I could think of (which is why I listed my brew history). – Handout

Amusingly enough the last answer from the last link is exactly what I was going to suggest: Forget messing around with pyenv, homebrew, and the system's Python install, and use Conda ;p – Komsomol

Yeah I'm a long time conda user trying to migrate to pyenv, as it seems to better suit my needs... if I can get it to work 😅 – Handout

Yeah I'm a long time conda user trying to migrate to pyenv, as it seems to better suit my needs That's certainly unexpected, can you elaborate? – Komsomol

@Komsomol I had to head out so I missed the chat. Sorry for bailing on you. I ended up figuring my issue. – Handout

@Komsomol Wanted to say thanks for your help—always appreciated! – Handout

You’re welcome, glad you figured things out :) – Komsomol

Changing Python version to 3.7 worked for me – Wife

Answer 1

None of the prior answers worked for me. The instructions in this gist worked for me.

In short: You may be missing the xz libraries, in which case you can install them with Homebrew and then reinstall/rebuild the Python version from pyenv:

$  brew install xz
$  pyenv uninstall <desired-python-version>
$  pyenv install <desired-python-version>

Note: I only had this problem with the Python installed by pyenv, but not the Mac system Python or the conda python. It might be best to use the brewed python (brew install python) unless you have a specific need for pyenv (like needing more control over the python version/updating).

Palomino

This worked for me as well. Installing xz was the secret sauce. Thanks!!! – Castorina

this worked for me: ie brew install xz then pyenv install 3.10.0 in the terminal – Hight

This solved the issue when installing python 3.10.9 on an M1 Mac via pyenv – Legere

Solved the issue, M1 Mac via pyenv. – Oxysalt

Still relevant when installing python versions with pyenv on M2 Macbook. Thanks!! – Castorina

This answer solved the issues on MacBook M2 as well. Thanks! – Adiell

Am confirming as this works for M2 Mac as well. Kudos! – Brandonbrandt

Works for M2 Pro - MacBook 2023 16". – Bruno

work perfectly on M2 16" Macbook Pro – Foreplay

This is indeed the issue when using pyenv on mac, with nothing preinstalled. Thank you. I believe this should be the chosen answer as it directly addresses the "missing .. library" error. – Mcintosh

Answer 2

Based on pyenv wiki, you should install the desired python version with --enable-framework flag. It did work for me.

PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install 3.6.7

The reason is that the module you're trying to use is implemented in C, and CPython with shared libs will be built with the aforementioned flag.

Picaroon

This is what I needed! Thank you so much!!! – Wayne

This did the trick, thanks ! In case you already have a pyenv python installed, no need to uninstall first. – Pottage

There appear to be issues with some versions of Python and/or OS: github.com/pyenv/pyenv/issues/99 – Corked

Answer 3

This solved it for me:

sudo apt-get install lzma
sudo apt-get install liblzma-dev
sudo apt-get install libbz2-dev

And add this files to your current environment: e.g. (see: https://github.com/pandas-dev/pandas/issues/27532#issuecomment-822272237):

sudo cp /usr/lib/python3.8/lib-dynload/_bz2.cpython-38-x86_64-linux-gnu.so /usr/local/lib/python3.8/
sudo cp /usr/lib/python3.8/lib-dynload/_lzma.cpython-38-x86_64-linux-gnu.so /usr/local/lib/python3.8/

For example, in my runtime environment:

sudo cp /usr/lib/python3.8/lib-dynload/_bz2.cpython-38-x86_64-linux-gnu.so /home/luca/.asdf/installs/python/3.8.10/lib/python3.8`
sudo cp /usr/lib/python3.8/lib-dynload/_lzma.cpython-38-x86_64-linux-gnu.so /home/luca/.asdf/installs/python/3.8.10/lib/python3.8

Billi

This one work for me, But rather than copy manually the libs, I rebuilt Python after installing lzma. – Twopiece

I was using Python3.7. I just had to do the 3 sudo commands provided and the comment from @Miguel Angel Retamozo Sanchez – Makowski

Answer 4

Google always lead me to this thread, but a good answer is in here

Briefly, you need to do:

CFLAGS="-I$(brew --prefix xz)/include" LDFLAGS="-L$(brew --prefix xz)/lib" pyenv install 3.9.1

If you are a M1 user this will work for you under rosetta.

framework build does not work for me.

Silly

thanks! I think this is essentially the same solution outlined here: https://mcmap.net/q/224824/-modulenotfounderror-no-module-named-39-_lzma-39-when-building-python-using-pyenv-on-macos – Handout

Answer 5

I solved this on Debian Buster by running:

sudo apt install liblzma-dev

After that, I recompiled Python.

I'm not an expert with Python, but it appears to have worked.

Importunacy

Answer 6

Ended up figuring out. This issue only arose when moving the Homebrew directory from its default. This was not needed when Homebrew was installed normally.

I added this to my .zshrc (alternatively, your .bashrc or .bash_profile)

export LDFLAGS="-L/Users/pcosta/.brew/opt/xz/lib $LDFLAGS"
export CPPFLAGS="-I/Users/pcosta/.brew/opt/xz/include $CPPFLAGS"
export PKG_CONFIG_PATH="/Users/pcosta/.brew/opt/xz/lib/pkgconfig:$PKG_CONFIG_PATH"

Homebrew warns that you should do this for other installed packages, but not xz. Presumably, because it is not needed if Homebrew lives where it expects to.

Handout

Doesn't seem to work my end. I tried in the terminal to run the export above before adding into the .zshrc file without success. Any other hint? – Moor

I also have these environmental variables—homebrew tells you to set them as you install the packages I listed above (openssl, readline, etc). I've had this issue again on CentOS, and again it had to do with xz. So that seems to be the main culprit here. A good first start is to make sure you install the needed packages listed in the wiki page., then make sure the necessary xz related packages are installed. Are you macOS? – Handout

Thanks I've solve a moment ago, and I didn't realised you answered. I believe the culprit here is the zlib that has to be installed on top of the xz. I can't confirm for sure as I did install a bunch of other things at the same time python brew install automake autoconf libtool pkg-config python zlib freetype libxml2 libxslt xz libyaml . But I also later run again a pyenv install xxxx so to foce a new make which eventually did solve the issue. – Moor

This automatically finds the right brew prefix export LDFLAGS="-L$(brew --prefix xz)/lib $LDFLAGS"; export CPPFLAGS="-I$(brew --prefix xz)/include $CPPFLAGS"; export PKG_CONFIG_PATH="$(brew --prefix xz)/lib/pkgconfig:$PKG_CONFIG_PATH" – Diplomatics

Answer 7

Issue 39430?

I don't have an answer. But this symptom sounds like this Python bug ticket:

• Issue 39430_ tarfile.open(mode=r) race condition when importing lzma - Python tracker

Reported in 2020. Unfortunately still unfixed as of 2022.

Workaround for package "csvkit": Try python2

I experienced this issue when trying to use package csvkit on my Windows 10 running MobaXterm:

Package installs just fine but then the /bin/csv* tools don't run:

$ /bin/python3 -m pip install csvkit --quiet
✔

$ csvstat --version
Traceback (most recent call last):
  File "/bin/csvstat", line 5, in <module>
    from csvkit.utilities.csvstat import launch_new_instance
  File "/usr/lib/python3.6/site-packages/csvkit/utilities/csvstat.py", line 12, in <module>
    from csvkit.cli import CSVKitUtility, parse_column_identifiers
  File "/usr/lib/python3.6/site-packages/csvkit/cli.py", line 16, in <module>
    import lzma
  File "/usr/lib/python3.6/lzma.py", line 27, in <module>
    from _lzma import *
ModuleNotFoundError: No module named '_lzma'
✘

My workaround was to use python2. Not super happy with that.

First throw out the non-working python3 version like so:

$ /bin/python3 -m pip uninstall csvkit --quiet --yes
✔

$ hash -r
✔

$ csvstat --version
csvstat: command not found
✘

So now it's gone. Let's try again but with python2:

$ /bin/python2 -m pip list | grep -i csv
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
WARNING: You are using pip version 20.3.4; however, version 21.3.1 is available.
You should consider upgrading via the '/bin/python2 -m pip install --upgrade pip' command.
✘

$ /bin/python2 -m pip install csvkit --quiet
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
  WARNING: The scripts csvclean, csvcut, csvformat, csvgrep, csvjoin, csvjson, csvlook, csvpy, csvsort, csvsql, csvstack, csvstat, in2csv and sql2csv are installed in '/usr/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: You are using pip version 20.3.4; however, version 21.3.1 is available.
You should consider upgrading via the '/bin/python2 -m pip install --upgrade pip' command.
✔

$ csvstat --version
csvstat 1.0.7
✔

Result: csvkit is installed and /bin/csvstat.exe runs without the "_lzma" error. But unfortunately on python2 and not on python3. -- This workaround is kinda good enough for me, because I just wanted the /bin/csv* utilities but maybe useless for people who actually need to run things on python3.

Triennial

This worked for me. I had to install python2 and pip2 and then I was able to install csvkit with python2 and it worked as describe above. Thank you. – Tensive

Answer 8

A lot of the answers here require brew for Mac development. If you don't have brew, such as when your development environment is managed by nix, you can do the following:

• Download the xz source from https://tukaani.org/xz/.
• Unzip it to a directory.
• Run ./configure --prefix=~/xz-install within the directory (you can change the prefix to be wherever you want to be). You may have to make sure the directory exists with mkdir -p beforehand.
• Run make within the directory (note: you will need xcode command line tools to compile this).
• Run make install. This will install to whatever directory you have set with --prefix.
• Assuming your --prefix directory is set to ~/xz-install/, install your python with CFLAGS="-I~/xz-install/include" LDFLAGS="-L~/xz-install/lib" pyenv install 3.10.4 or whatever version you want.

Daigle

Answer 9

Solution

frist

yum install xz-devel
yum install python-backports-lzma
pip install backports.lzma

second

cd /usr/local/lib/python3.8
vi lzma.py 

finally modify about line 27

try:
    from _lzma import *
    from _lzma import _encode_filter_properties, _decode_filter_properties
except ImportError:
    from backports.lzma import *
    from backports.lzma import _encode_filter_properties, _decode_filter_properties

it‘s work！I had to search for hours to get this problem to be solved.

Credit: Yolo5 issues

Cheliform

Answer 10

If you installed python via asdf, you can reinstall Python like so:

asdf uninstall python

asdf install python latest

Example output of the install command:

python-build 3.10.6 /Users/u/.asdf/installs/python/3.10.6
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.6.tar.xz...
-> https://www.python.org/ftp/python/3.10.6/Python-3.10.6.tar.xz
Installing Python-3.10.6...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.10.6 to /Users/u/.asdf/installs/python/3.10.6

Relapse

Answer 11

The solution in Ubuntu 22.04.3 LTS on a commodity PC is:

sudo apt install libpython3.10-stdlib libpython3.10-testsuite pypy3-lib

If you don't want to fill your disk with trash, the first library must work.

Aragonite

Answer 12

Error Message

>>> import lzma
File "/usr/local/lib/python3.5/lzma.py", line 27, in <module>
    from _lzma import *
ModuleNotFoundError: No module named '_lzma'

Solution

Uninstall all versions of Python3. Then reinstall only python3.5.

Original Environment (Environment that has error)

• Linux 4.4.0-62-generic
• Python2.7
• Python3.4
• Python3.5
• Python3.6
• Python3.8

Current Environment (No error, everything works well)

• Linux 4.4.0-62-generic
• Python2.7
• Python3.5

Coxcombry