_C.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceIdEEPKNS_6detail12TypeMetaDataEv

Question

What is the reason for this error and how can I fix it? I am running the code from this repo: https://github.com/facebookresearch/frankmocap

(frank) mona@goku:~/research/code/frankmocap$ python -m demo.demo_frankmocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/mona/research/code/frankmocap/demo/demo_frankmocap.py", line 25, in <module>
    from handmocap.hand_bbox_detector import HandBboxDetector
  File "/home/mona/research/code/frankmocap/handmocap/hand_bbox_detector.py", line 33, in <module>
    from detectors.hand_object_detector.lib.model.roi_layers import nms # might raise segmentation fault at the end of program
  File "/home/mona/research/code/frankmocap/detectors/hand_object_detector/lib/model/roi_layers/__init__.py", line 3, in <module>
    from .nms import nms
  File "/home/mona/research/code/frankmocap/detectors/hand_object_detector/lib/model/roi_layers/nms.py", line 3, in <module>
    from model import _C
ImportError: /home/mona/research/code/frankmocap/detectors/hand_object_detector/lib/model/_C.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceIdEEPKNS_6detail12TypeMetaDataEv

I have:

$ lsb_release -a
LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:    20.04
Codename:   focal

and

$ python
Python 3.8.5 (default, Jan 27 2021, 15:41:15) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.8.1+cu111'
>>> import detectron2
>>> detectron2.__version__
'0.4'
>>> from detectron2 import _C

and:

$ python -m detectron2.utils.collect_env
/home/mona/venv/frank/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0
No CUDA runtime is found, using CUDA_HOME='/usr'
---------------------  --------------------------------------------------------------------------
sys.platform           linux
Python                 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]
numpy                  1.19.5
detectron2             0.4 @/home/mona/venv/frank/lib/python3.8/site-packages/detectron2
Compiler               GCC 7.3
CUDA compiler          CUDA 11.1
DETECTRON2_ENV_MODULE  <not set>
PyTorch                1.8.1+cu111 @/home/mona/venv/frank/lib/python3.8/site-packages/torch
PyTorch debug build    False
GPU available          False
Pillow                 8.1.0
torchvision            0.9.1+cu111 @/home/mona/venv/frank/lib/python3.8/site-packages/torchvision
fvcore                 0.1.3.post20210311
cv2                    4.5.1
---------------------  --------------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

Britt

Hello @Mona, How did you fix the issue? Thanks. – Humane

Does that mean your PyTorch version doesn't go well with your environment? I used torch1.7+cu110 to circumvent this issue. – Pack

Answer 1

This error usually shows up when there's a compatibility issue between the installed pytorch version and the detector library version(Detectron2 or mmdet).

Both the detector library and pytorch have to be built by the same CUDA version otherwise some packages will conflict when training your model.

There's a possibility that the Pytorch (1.8.1) + CUDA version (11.1) you have is incompatible with detectron2 v(0.4).

From the repo here, detectron2 v0.4 is built with torch 1.8 + cuda 11.1. It might help if you use torch 1.8.0 instead of 1.8.1

Throstle

I tried with Pytorch 1.8.0 but still the issue didn't get fixed. – Pandean

Answer 2

I tried the solutions mentioned here, but that didn't fully solve the problem. However, when I tried solving a different error using this solution, it also solved this error for me. Use the following command:

pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

Chasse

I tried with Pytorch 1.8.0 but still the issue didn't get fixed. Any guess? – Pandean

Answer 3

well, i got the problem too. and i fixed it. it means that the static library /home/mona/research/code/frankmocap/detectors/hand_object_detector/lib/model/_C.cpython-38-x86_64-linux-gnu.so is not appropriate to this program. i guess you reinstalled something such as torch or detectron2.

to fix this problem, try to reinstall detectron2 and remove the detectors directory in frankmocap, then use ./scripts/install_frankmocap.sh ./scripts/install_hand_detectors.sh and ./scripts/install_pose2d.sh to reconstruct libraries which needed by frankmocap.

after all, maybe you need copy the _C.cpython-38-x86_64-linux-gnu.so from detectors/hand_object_detector/lib/build/lib.linux-x86_64-3.6/model to detectors/hand_object_detector/lib/model. maybe not

Philanthropist

Answer 4

I've faced of this issue recently and creating a new environment with parameters below solved my problem.

PS : I assume that you're using torch with CUDA, if you're not you can install non-GPU versions.

torch version : 1.9 + cu111
torchvision : 0.10.0 + cu111
torchaudio : 0.9.0

You can download these from ;

pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f 

For Detectron2 ;

python -m pip install detectron2 -f \
  https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.9/index.html

https://download.pytorch.org/whl/torch_stable.html

Source for PyTorch previous versions link

Lorelle