Installing libmagic with pip fails

Question

After installing in my Jupyter Notebook (as a container of JupyterLab as jovan user without access to root) the libmagic while having cmake 3.26.4 already installed in the conda env. I try to install install libmagic with pip:

pip install python-libmagic

but I keep getting error:

Collecting python-libmagic
  Using cached python_libmagic-0.4.0-py3-none-any.whl
Collecting cffi==1.7.0 (from python-libmagic)
  Using cached cffi-1.7.0.tar.gz (400 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: pycparser in /opt/conda/envs/cho_env/lib/python3.10/site-packages (from cffi==1.7.0->python-libmagic) (2.21)
Building wheels for collected packages: cffi
  Building wheel for cffi (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [254 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/verifier.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/commontypes.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/recompiler.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/cparser.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/lock.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/__init__.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/model.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/api.py -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-x86_64-cpython-310/cffi
      copying cffi/_embedding.h -> build/lib.linux-x86_64-cpython-310/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/c
      gcc -pthread -B /opt/conda/envs/cho_env/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/conda/envs/cho_env/include -fPIC -O2 -isystem /opt/conda/envs/cho_env/include -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/opt/conda/envs/cho_env/include/python3.10 -c c/_cffi_backend.c -o build/temp.linux-x86_64-cpython-310/c/_cffi_backend.o
      In file included from c/_cffi_backend.c:274:
      c/minibuffer.h: In function ‘mb_ass_slice’:
      c/minibuffer.h:66:5: warning: ‘PyObject_AsReadBuffer’ is deprecated [-Wdeprecated-declarations]
         66 |     if (PyObject_AsReadBuffer(other, &buffer, &buffer_len) < 0)
            |     ^~
      In file included from /opt/conda/envs/cho_env/include/python3.10/genobject.h:12,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:110,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/abstract.h:343:17: note: declared here
        343 | PyAPI_FUNC(int) PyObject_AsReadBuffer(PyObject *obj,
            |                 ^~~~~~~~~~~~~~~~~~~~~
      In file included from c/_cffi_backend.c:277:
      c/file_emulator.h: In function ‘PyFile_AsFile’:
      c/file_emulator.h:54:14: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
         54 |         mode = PyText_AsUTF8(ob_mode);
            |              ^
      In file included from c/_cffi_backend.c:281:
      c/wchar_helper.h: In function ‘_my_PyUnicode_AsSingleWideChar’:
      c/wchar_helper.h:83:5: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
         83 |     Py_UNICODE *u = PyUnicode_AS_UNICODE(unicode);
            |     ^~~~~~~~~~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
        580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
            |                                             ^~~~~~~~~~~~~~~~~~~
      In file included from c/_cffi_backend.c:281:
      c/wchar_helper.h:84:5: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
         84 |     if (PyUnicode_GET_SIZE(unicode) == 1) {
            |     ^~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
        446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
            |                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from c/_cffi_backend.c:281:
      c/wchar_helper.h:84:5: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
         84 |     if (PyUnicode_GET_SIZE(unicode) == 1) {
            |     ^~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
        580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
            |                                             ^~~~~~~~~~~~~~~~~~~
      In file included from c/_cffi_backend.c:281:
      c/wchar_helper.h:84:5: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
         84 |     if (PyUnicode_GET_SIZE(unicode) == 1) {
            |     ^~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
        446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
            |                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from c/_cffi_backend.c:281:
      c/wchar_helper.h: In function ‘_my_PyUnicode_SizeAsWideChar’:
      c/wchar_helper.h:99:5: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
         99 |     Py_ssize_t length = PyUnicode_GET_SIZE(unicode);
            |     ^~~~~~~~~~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
        446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
            |                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from c/_cffi_backend.c:281:
      c/wchar_helper.h:99:5: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
         99 |     Py_ssize_t length = PyUnicode_GET_SIZE(unicode);
            |     ^~~~~~~~~~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
        580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
            |                                             ^~~~~~~~~~~~~~~~~~~
      In file included from c/_cffi_backend.c:281:
      c/wchar_helper.h:99:5: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
         99 |     Py_ssize_t length = PyUnicode_GET_SIZE(unicode);
            |     ^~~~~~~~~~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
        446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
            |                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
      In file included from c/_cffi_backend.c:281:
      c/wchar_helper.h: In function ‘_my_PyUnicode_AsWideChar’:
      c/wchar_helper.h:118:5: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
        118 |     Py_UNICODE *u = PyUnicode_AS_UNICODE(unicode);
            |     ^~~~~~~~~~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
        580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
            |                                             ^~~~~~~~~~~~~~~~~~~
      c/_cffi_backend.c: In function ‘ctypedescr_dealloc’:
      c/_cffi_backend.c:352:23: error: lvalue required as left operand of assignment
        352 |         Py_REFCNT(ct) = 43;
            |                       ^
      c/_cffi_backend.c:355:23: error: lvalue required as left operand of assignment
        355 |         Py_REFCNT(ct) = 0;
            |                       ^
      c/_cffi_backend.c: In function ‘cast_to_integer_or_char’:
      c/_cffi_backend.c:3331:26: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
       3331 |                          PyUnicode_GET_SIZE(ob), ct->ct_name);
            |                          ^~~~~~~~~~~~~~~~~~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
        446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
            |                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
      c/_cffi_backend.c:3331:26: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
       3331 |                          PyUnicode_GET_SIZE(ob), ct->ct_name);
            |                          ^~~~~~~~~~~~~~~~~~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here
        580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
            |                                             ^~~~~~~~~~~~~~~~~~~
      c/_cffi_backend.c:3331:26: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations]
       3331 |                          PyUnicode_GET_SIZE(ob), ct->ct_name);
            |                          ^~~~~~~~~~~~~~~~~~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here
        446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) {
            |                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
      c/_cffi_backend.c: In function ‘b_complete_struct_or_union’:
      c/_cffi_backend.c:4251:17: warning: ‘PyUnicode_GetSize’ is deprecated [-Wdeprecated-declarations]
       4251 |                 do_align = PyText_GetSize(fname) > 0;
            |                 ^~~~~~~~
      In file included from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:177:43: note: declared here
        177 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_ssize_t) PyUnicode_GetSize(
            |                                           ^~~~~~~~~~~~~~~~~
      c/_cffi_backend.c:4283:13: warning: ‘PyUnicode_GetSize’ is deprecated [-Wdeprecated-declarations]
       4283 |             if (PyText_GetSize(fname) == 0 &&
            |             ^~
      In file included from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:177:43: note: declared here
        177 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_ssize_t) PyUnicode_GetSize(
            |                                           ^~~~~~~~~~~~~~~~~
      c/_cffi_backend.c:4353:17: warning: ‘PyUnicode_GetSize’ is deprecated [-Wdeprecated-declarations]
       4353 |                 if (PyText_GetSize(fname) > 0) {
            |                 ^~
      In file included from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:177:43: note: declared here
        177 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_ssize_t) PyUnicode_GetSize(
            |                                           ^~~~~~~~~~~~~~~~~
      c/_cffi_backend.c: In function ‘prepare_callback_info_tuple’:
      c/_cffi_backend.c:5214:5: warning: ‘PyEval_InitThreads’ is deprecated [-Wdeprecated-declarations]
       5214 |     PyEval_InitThreads();
            |     ^~~~~~~~~~~~~~~~~~
      In file included from /opt/conda/envs/cho_env/include/python3.10/Python.h:130,
                       from c/_cffi_backend.c:2:
      /opt/conda/envs/cho_env/include/python3.10/ceval.h:122:37: note: declared here
        122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
            |                                     ^~~~~~~~~~~~~~~~~~
      c/_cffi_backend.c: In function ‘b_callback’:
      c/_cffi_backend.c:5255:5: warning: ‘ffi_prep_closure’ is deprecated: use ffi_prep_closure_loc instead [-Wdeprecated-declarations]
       5255 |     if (ffi_prep_closure(closure, &cif_descr->cif,
            |     ^~
      In file included from c/_cffi_backend.c:15:
      /opt/conda/envs/cho_env/include/ffi.h:347:1: note: declared here
        347 | ffi_prep_closure (ffi_closure*,
            | ^~~~~~~~~~~~~~~~
      In file included from /opt/conda/envs/cho_env/include/python3.10/unicodeobject.h:1046,
                       from /opt/conda/envs/cho_env/include/python3.10/Python.h:83,
                       from c/_cffi_backend.c:2:
      c/ffi_obj.c: In function ‘_ffi_type’:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:744:29: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
        744 | #define _PyUnicode_AsString PyUnicode_AsUTF8
            |                             ^~~~~~~~~~~~~~~~
      c/_cffi_backend.c:72:25: note: in expansion of macro ‘_PyUnicode_AsString’
         72 | # define PyText_AS_UTF8 _PyUnicode_AsString
            |                         ^~~~~~~~~~~~~~~~~~~
      c/ffi_obj.c:191:32: note: in expansion of macro ‘PyText_AS_UTF8’
        191 |             char *input_text = PyText_AS_UTF8(arg);
            |                                ^~~~~~~~~~~~~~
      c/lib_obj.c: In function ‘lib_build_cpython_func’:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:744:29: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
        744 | #define _PyUnicode_AsString PyUnicode_AsUTF8
            |                             ^~~~~~~~~~~~~~~~
      c/_cffi_backend.c:72:25: note: in expansion of macro ‘_PyUnicode_AsString’
         72 | # define PyText_AS_UTF8 _PyUnicode_AsString
            |                         ^~~~~~~~~~~~~~~~~~~
      c/lib_obj.c:129:21: note: in expansion of macro ‘PyText_AS_UTF8’
        129 |     char *libname = PyText_AS_UTF8(lib->l_libname);
            |                     ^~~~~~~~~~~~~~
      c/lib_obj.c: In function ‘lib_build_and_cache_attr’:
      /opt/conda/envs/cho_env/include/python3.10/cpython/unicodeobject.h:744:29: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
        744 | #define _PyUnicode_AsString PyUnicode_AsUTF8
            |                             ^~~~~~~~~~~~~~~~
      c/_cffi_backend.c:71:24: note: in expansion of macro ‘_PyUnicode_AsString’
         71 | # define PyText_AsUTF8 _PyUnicode_AsString   /* PyUnicode_AsUTF8 in Py3.3 */
            |                        ^~~~~~~~~~~~~~~~~~~
      c/lib_obj.c:208:15: note: in expansion of macro ‘PyText_AsUTF8’
        208 |     char *s = PyText_AsUTF8(name);
            |               ^~~~~~~~~~~~~
      In file included from c/cffi1_module.c:16,
                       from c/_cffi_backend.c:6636:
      c/lib_obj.c: In function ‘lib_getattr’:
      c/lib_obj.c:506:7: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
        506 |     p = PyText_AsUTF8(name);
            |       ^
      In file included from c/cffi1_module.c:19,
                       from c/_cffi_backend.c:6636:
      c/call_python.c: In function ‘_get_interpstate_dict’:
      c/call_python.c:20:30: error: dereferencing pointer to incomplete type ‘PyInterpreterState’ {aka ‘struct _is’}
         20 |     builtins = tstate->interp->builtins;
            |                              ^~
      c/call_python.c: In function ‘_ffi_def_extern_decorator’:
      c/call_python.c:73:11: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
         73 |         s = PyText_AsUTF8(name);
            |           ^
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cffi
  Running setup.py clean for cffi
Failed to build cffi
ERROR: Could not build wheels for cffi, which is required to install pyproject.toml-based projects```

how can I fix this?

Palenque

Answer 1

Correct me, if I'm wrong, but I guess you want to install the python bindings for libmagic.

Find them here: https://pypi.org/project/python-magic/ and here https://github.com/ahupp/python-magic

pip install python-magic

There was an issue request 6 years ago by its maintainer to the person sitting on the abandoned "python-libmagic" name to change it, to no avail.

The pip package "libmagic" from here https://pypi.org/project/libmagic/ is an even older bindings (basically a python interface to the library) and not the same as the library either. This package looks abandoned as well.

And this abandoned pip package seems to interfere with the conda package that you need to install, so:

pip uninstall libmagic

And with conda install the actual library like you did:

conda install -c conda-forge libmagic

Just for reference in other cases, one can install system packages for libmagic, for instance:

apt install libmagic1 libmagic-dev

Turbofan

@sm I use Ubuntu JupyterLab and I tried to install it as I keep getting this error libmagic is unavailable but assists in filetype detection on file-like objects. Please consider installing libmagic for better results. – Palenque

@Palenque Did you try pip install python-magic-bin as well? And was libmagic1 installed? – Turbofan

@sm yes, I kep gettng the error: ERROR: Could not find a version that satisfies the requirement python-magic-bin (from versions: none) ERROR: No matching distribution found for python-magic-bin And the lipi list returns libmagic 1.0 – Palenque

Sorry for the confusion, but did you install (apt install libmagic1 libmagic-dev ) before? pip install libmagic installs an even older bindings (pypi.org/project/libmagic) which is not the same as the library you need. (You can remove it, keep python-magic and install libmagic1 libmagic-dev as mentioned above) – Turbofan

And it looks like there is also a conda for it: anaconda.org/conda-forge/libmagic – Turbofan

I've installed it with conda install -c conda-forge libmagic. It has installed it like: libmagic conda-forge/linux-64::libmagic-5.39-h753d276_1 but when I run: 'conda list | grep magic libmagic 1.0 pypi_0 pypi pip-magic 0.2.3 pypi_0 pypi python-magic 0.4.27 pypi_0 pypi ` So it is version 1.0 and not the one i've installed 5.39.. wired. – Palenque

I think libmagic 1.0 is still the bindings (which is only the connector to libmagic) that you installed with pip. Those you can anyway get rid of with pip uninstall libmagic. Maybe it helps to first uninstall the non-needed package (libmagic 1.0), make sure it is gone and then try to install the conda-forge libmagic – Turbofan

Yes it worked. Uninstalling pip uninstall libmagic made the new version to came up and it works. PLeaase add it to your answer so that I can accept :) – Palenque

Nice job :) I added it to the answer. – Turbofan

I;m not able to install via conda-forge conda install -c conda-forge libmagic PackagesNotFoundError: The following packages are not available from current channels – Contractile