CMake project fails to find shared library

Question

I am using CMake to build a cross platform project. For the moment I am trying to run it on Linux. I have recently added a project for running tests, but it will not run because it cannot find one of the shared libraries, specifically libtbbmalloc.so.2:

/tests: error while loading shared libraries: libtbbmalloc.so.2: cannot open shared object file: No such file or directory`

When I run ldd on the executable I get the following:

linux-vdso.so.1 (0x00007fffeb572000)
libtbbmalloc_proxy.so.2 => /home/username/dev/tbb/libtbbmalloc_proxy.so.2 (0x00007f50afe00000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f50afa70000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f50af6d0000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f50af4b0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f50af0a0000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f50aee90000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f50aec70000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f50aea60000)
/lib64/ld-linux-x86-64.so.2 (0x00007f50b0400000)
libtbbmalloc.so.2 => not found

The CMakeLists.txt for my test project looks like this:

set(test_sourcefiles main_tests.cpp)

add_executable(tests ${test_sourcefiles})

target_link_libraries(tests Catch2::Catch2 MyLib)

MyLib uses tbb, and I guess that is why my executable (tests) searches for it. When running ldd on MyLib it finds the library (libtbbmalloc.so.2):

(removed some output for readability)
libtbbmalloc_proxy.so.2 => /home/username/dev/tbb/libtbbmalloc_proxy.so.2 (0x00007f9af8110000)
libtbbmalloc.so.2 => /home/username/dev/tbb/libtbbmalloc.so.2 (0x00007f9ac4eb0000)

I have tried specifically adding libttbbmalloc.so.2 in my tests/CMakeLists.txt target_link_libraries(${project} /home/username/dev/tbb/libtbbmalloc.so.2), but it makes no difference.

If I add /home/username/dev/tbb/ to LD_LIBRARY_PATH, the program runs, and ldd reports that libtbbmalloc.so.2 is found.

Any ideas on what I might be doing wrong, and how can I get my program to run without setting LD_LIBRARY_PATH?

Update: I found out that it's possible to print runpath/rpath by using chrpath -l name-of-executable. When using this tool on my executable, it looks like the folder with libtbbmalloc.so.2 is added to runpath, but the program still won't run:

larjr@DESKTOP:~/dev/project/build/tests$ chrpath -l tests
tests: RUNPATH=/home/larsjr/dev/project/build/MyLib:/home/username/dev/tbb

Power

All your ldd commands show that your library is correctly resolved (found). So if you run your tests at the same build location, it shoud work. The answer by @Zaffy assumes that your problem is after the installation of your program. Is that the problem? – Notarize

No, the problem does not occur after installing the program. For now I'm trying to run it in the build location. The output from the first ldd command (should perhaps have highlighted this) shows that it cannot resolve libtbbmalloc.so.2 . when building the test project, right? What I do not understand is why one project MyLib, works correctly, while my tests project does not. – Power

You are right, sorry. Your first ldd output says that libtbbmalloc_proxy.so.2 is correctly resolved, but libtbbmalloc.so.2 is not! And that explains it. Your program links to the proxy library, so the RPATH of your program is used to locate this one. But the real library is not linked to your program, it is linked to the proxy library. And the proxy library does not have an embedded RPATH to help the linker to locate the buddy. – Notarize

That makes sense. Any suggestions for what I can do to make the proxy library find libtbbmalloc.so.2? I've tried explicitly linking against it in my tests project, but it still cannot be found. – Power

I would try two options: 1: patchelf the proxy library to insert a RPATH on it, or 2: simply install the libraries on your system at some standard prefix like /usr/local/ that is included in your /etc/ld.so.conf – Notarize

Answer 1

The issue you are encountering does sound to be related to the search path for direct and indirect linkage of shared libraries. CMake may build with the new ld RUNPATH variable which will only work on direct dependents, where RPATH will work recursively on all indirect dependents. As shown in the snippet at bottom of this post, this may depend on your version of GCC. To check if your executable is using RPATH or RUNPATH, run:

$ readelf -d ./executable

and find the library rpath / runpath line after your dependencies. ie/eg: It may show either of the below:

a) 0x000000000000001d (RUNPATH)            Library runpath: [/some/dir/lib]
b) 0x000000000000001d (RPATH)              Library rpath:   [/some/dir/lib]

You showed you used the command chrpath -l ./executable, which actually spat out:

larjr@DESKTOP:~/dev/project/build/tests$ chrpath -l tests

tests: RUNPATH=/home/larsjr/dev/project/build/MyLib:/home/username/dev/tbb

It says here "RUNPATH" which is non-recursive - that's your error.

Refer to this excellent github issue where members are discussing indirect linking issues with CMake, highly suggest reading it all. It has a a great example regarding dependencies and their search paths. https://github.com/conan-io/conan/issues/2660

A snippet regarding new vs old linker behavior which can cause this 'issue':

The problem you are experiencing in conan-deptest (excellent isolated case, by the way!) is the behaviour that you are documenting, between DT_RPATH and DT_RUNPATH. It is CMake (and not conan), that passes the -rpath flag with the correct paths to the dependent libraries, when building the executable. The problem is that newer versions of the linker in some platforms, emit DT_RUNPATH where they used to emit DT_RPATH. Indeed this is the case as readelf -d is listing the RUNPATH tag.

There is also another stackoverflow post where a user asks about this change in behaviour in LD/GCC: How to set RPATH and RUNPATH with GCC/LD?

A conan member further elaborates on how to change this behavior:

The way to replicate this behaviour on versions of the linker that emit RUNPATH instead, is to pass the --disable-new-dtags flag to the linker.

And the resulting cmake:

target_link_libraries(target "-Wl,--disable-new-dtags")

Now if you rebuild, and check readelf -d on your executable, you should see the RPATH tag. You can run:

env LD_DEBUG=files,libs ./executable

to see RPATH being passed into the search path for each dependency:

31658:  file=libbar.so.1 [0];  needed by /home/user/projectA/lib/libfoo.so.2.5 [0]
31658:  find library=libbar.so.1 [0]; searching
31658:   search path=/home/user/projectA/lib        (RPATH from file ./executable)
31658:    trying file=/home/user/projectA/lib/libbar.so.1

Hopefully this fixes your issue!

Bonus: Link to tutorial on shared libraries. At the bottom of the aritcle shows runpath vs rpath + origin! https://amir.rachum.com/blog/2016/09/17/shared-libraries/

Naji

Answer 2

I believe you are looking for setting a runtime path on the executable:

set_target_properties(tests PROPERTIES
    INSTALL_RPATH "<your path to libs>"
    BUILD_WITH_INSTALL_RPATH 1
)

INSTALL_RPATH

Normally CMake uses the build tree for the RPATH when building executables etc on systems that use RPATH. When the software is installed the executables etc are relinked by CMake to have the install RPATH. If this variable is set to true then the software is always built with the install path for the RPATH and does not need to be relinked when installed.

BUILD_WITH_INSTALL_RPATH

Normally CMake uses the build tree for the RPATH when building executables etc on systems that use RPATH. When the software is installed the executables etc are relinked by CMake to have the install RPATH. If this variable is set to true then the software is always built with the install path for the RPATH and does not need to be relinked when installed.

Dyun

This did unfortunately not solve my problem. I still get the same error message. I found a command to print rpath, and it seems like the correct folder is added to the path, but it still can't find the library. The strange thing is that it seems like the correct it had the correct folders added to path without adding the command you suggest, but it still does not work. I'll update my question with details. – Power

@Power can you please run strace on your exe? – Dyun

I have run strace, but I'm not familiar with the tool, so the output doesn't tell me much yet. I uploaded it here: gist.github.com/larsjr/673ce1eb1887b132745f74be95925264 Note that in my original question I simplified some of the paths. The location of the folder with libtbbmalloc.so.2 is /home/lars/dev/mi-mil/build/mimil-build/packages/icgi-intelTBB.2019.8.281.1/build/native/lib/linuxx64/Release – Power

@Power strace hooks your program and displays all syscalls it makes. This is useful in your case because you can see where the linker is looking for the library. It looks like tbbmalloc_proxy was found just fine, but for lbbmalloc was not even searched in your rpath. Could mean tbbmalloc_proxy tries to load the other tbbmalloc on the fly? Would it be possible for you to make a "hello world" program to reproduce this for me? – Dyun

Sorry, I haven't been able to work the last few days. I have tried creating a "hello world" example, but I'm unable to reproduce the error. It seems that while simplifying, I'm losing some vital part. The example is uploaded here if it might be of any use: github.com/larsjr/tbb_link_example (when running ldd on libproject1.so it reports it as a "statically linked library"). – Power

Answer 3

add set_target_properties next to your executable install command

# install
INSTALL(TARGETS ${PROJECT_NAME} myLib
    RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
    LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
)

set_target_properties(${PROJECT_NAME} PROPERTIES
    INSTALL_RPATH ${CMAKE_INSTALL_PREFIX}/lib
    BUILD_WITH_INSTALL_RPATH ON
)

Bombastic