Gensim install in Python 3.11 fails because of missing longintrepr.h file

Question

Operating System: macOS Monterey 12.6 Chip: Apple M1 Python version: 3.11.1

I try:

pip3 install gensim

The install process starts well but fatally fails towards the end while running 'clang'. The error message is:

clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -I/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/numpy/core/include -c gensim/models/word2vec_inner.c -o build/temp.macosx-10.9-universal2-cpython-311/gensim/models/word2vec_inner.o
      gensim/models/word2vec_inner.c:217:12: fatal error: 'longintrepr.h' file not found
        #include "longintrepr.h"
                 ^~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

This issue is raised in a couple of github postings and is attributed to some incompatibility between cython and python 3.11. However, no suggestion is forwarded as to what users should do until cython is updated. I may have misrepresented the details of the discussions on github but I think this is the gist of it.

Can anyone help me in installing gensim in the meantime?

Thanks.

I updated cython and aiohttp. The latter because I had seen a post where the aiohttp install failed for the same reason as mine (missing "longintrepr.h").

No improvement. "pip install gensim" still fails and fails with the same message as copied above.

Metopic

Answer 1

I also faced the same issue for gensim library on Windows laptop while using Python 3.11.1 Changing to the Python 3.10 worked for me.

Apospory

I can't believe it, downgrading worked for me as well (using Linux Ubuntu) – Outplay

Answer 2

It seems your issue may be due to the specifics of a fairly-new Python, and lagging library support, on a somewhat new system (a MacOS M1 machine) which has its own somewhat-unique build toolchains.

Unless you absolutely need to use Python 3.11.1, I'd suggest using Gensim within a Python environment with a slightly-older Python interpreter, where the various pacakges you truly need may be a little more settled. For example, on many OS/architecture/Python combinations, a standard pip install will grab precompiled libraries – so build errors of the type you're seeing can't happen.

That your installation is falling back to a local compilation (which hits a problem without an easy off-the-shelf solution) is a hint that something about the full configuration is still somewhat undersupported by one or more of the involved libraries.

If you use the conda 3rd-party system for managing Python virtual environments, it also offers you the ability to explicitly choose which Python version will be used in each environment. That is, you're not stuck with the exact version, and installed libraries, that are default/global on your OS. You could easily try Python 3.10, or Python 3.9, which might work better.

And, keeping your development/project virtual-environment distinct from the system's Python is often considered a "best practice" for other purposes, too. There's no risk that you'll cause damage to the system Python and any tools reliant on it, or face an issue where several of your Python projects need conflicting library versions. (You just use a separate environment for eah project.) And, the exercise of rigorously specifying what needs to be in your project's environment helps keep its prerequisites/dependencies clear for any future relocations/installations elsewhere.

When using the conda tool for this purpose, I usually start with the miniconda version, so that I have explicit control over exactly what packages are installed, and can thus keep each environment minimally-specified for its purposes. (The larger anaconda approach pre-installs tons of popular packages instead.)

Martyry

Further to my first comment, pursuing advice from @gojomo, I installed a virtual environment in miniconda and was able import gensim when I used Python 3.8. The pip did not work but I installed using setup.py. – Metopic

Answer 3

I had a similar issue with Python3.11 while building Pycrypto on GNU/Linux. As a dirty workaround, I copied https://github.com/python/cpython/blob/main/Include/cpython/longintrepr.h to /usr/include/python3.11/longintrepr.h and it worked!

Kelton

Troubles building qmeq and this at least solved that error. with my 3.11.3 Python. – Rattlehead

This solved my issue installing bigfloat. – Reeding

Thanks, although it sure is ugly! This hack worked for me also to install beaker (which requires pycrypto) on a raspberry pi with 3.11. – Boride

This would only work if you're running your application locally - I don't think you would be able to deploy an application with this hack. – Cinque

copy header from your precise version seems better than copy from main stream – Gravitative

Answer 4

I pointed to the correct include location for python3.11. Downgrading isn’t necessary.

Correct method is:

#include <python3.11/cpython/longintrepr.h>             /* for conversions */ 

Fibro