Login/Register
Dynamic linking with rpath not working under Ubuntu 17.10
Asked Answered
Solved rmakefileg++rcppdynamic-linking
D

1

7

I build an R package which uses Rcpp and links to a third-party shared object (libbarraopt.so) (which also links to other shared objects such as liboptsrvr.so in its own directory). To ensure that it is able to find those shared objects it links against, I put the following variables in ~/.Renviron:

BARRA_OPS_HOME=${HOME}/bin/BarraOptimizer8.5

In the package, I create the following src/Makevars:

BARRA_LIB=$(BARRA_OPS_HOME)/lib/intel64
BARRA_INCLUDE=$(BARRA_OPS_HOME)/include
PKG_CXXFLAGS=-I$(BARRA_INCLUDE)
PKG_CFLAGS=-I$(BARRA_INCLUDE)
PKG_LIBS=-L$(BARRA_LIB) -Wl,-R,$(BARRA_LIB) -lbarraopt

Under Ubuntu 16.04, I can build, load, and use the package successfully without any problem. However, when I test exactly the same package when OS is upgraded to 17.10, the package can be built but it cannot be loaded, saying:

g++ -std=gnu++11 -I/usr/share/R/include -DNDEBUG  -I"/home/renkun/R/x86_64-pc-linux-gnu-library/3.4/Rcpp/include"   -I/home/renkun/bin/BarraOptimizer8.5/include -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c RcppExports.cpp -o RcppExports.o
** libs
g++ -std=gnu++11 -I/usr/share/R/include -DNDEBUG  -I"/home/renkun/R/x86_64-pc-linux-gnu-library/3.4/Rcpp/include"   -I/home/renkun/bin/BarraOptimizer8.5/include -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c barraopt.cpp -o barraopt.o
g++ -std=gnu++11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o barraopt.so RcppExports.o barraopt.o -L/home/renkun/bin/BarraOptimizer8.5/lib/intel64 -Wl,-R,/home/renkun/bin/BarraOptimizer8.5/lib/intel64 -lbarraopt -L/usr/lib/R/lib -lR
installing to /tmp/Rtmpvbb6Io/devtools_install_42a342a07f84/barraopt/libs
* DONE (barraopt)
Error in dyn.load(dllfile) : 
  unable to load shared object '/home/renkun/Workspaces/barraopt/src/barraopt.so':
  liboptsrvr.so: cannot open shared object file: No such file or directory
Calls: suppressPackageStartupMessages ... <Anonymous> -> load_all -> load_dll -> library.dynam2 -> dyn.load
Execution halted

Exited with status 1.

It seems that -Wl,-rpath is not effective here.

Under a machine with Ubuntu 16.04, ldd src/barraopt.so shows that all dynamic linking is corrected resolved. (BARRA_OPS_HOME = /home/ken/bin/BarraOptimizer8.5)

linux-vdso.so.1 =>  (0x00007ffc89a16000)
libbarraopt.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libbarraopt.so (0x00007f85dae49000)
libimf.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libimf.so (0x00007f85da97f000)
libR.so => /usr/lib/libR.so (0x00007f85da346000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f85d9fc4000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f85d9dae000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f85d99e3000)
liboptsrvr.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/liboptsrvr.so (0x00007f85d7b10000)
libopsproto.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libopsproto.so (0x00007f85d77a1000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f85d7497000)
libintlc.so.5 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libintlc.so.5 (0x00007f85d7249000)
libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f85d6fe8000)
libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007f85d6da1000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f85d6b31000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f85d690f000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f85d66fe000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f85d64e4000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f85d62dc000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f85d60d7000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f85d5eb5000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f85d5c98000)
/lib64/ld-linux-x86-64.so.2 (0x000055fb75088000)
libifcore.so.5 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libifcore.so.5 (0x00007f85d5961000)
libifport.so.5 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libifport.so.5 (0x00007f85d5732000)
libsvml.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libsvml.so (0x00007f85d4e6d000)
libmosek64.so.7.0 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libmosek64.so.7.0 (0x00007f85d3c63000)
libiomp5.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libiomp5.so (0x00007f85d396b000)
libprotobuf.so.6 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libprotobuf.so.6 (0x00007f85d3668000)
libbridge_common.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libbridge_common.so (0x00007f85d3417000)
libsharc_xmlxproto.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libsharc_xmlxproto.so (0x00007f85d31a4000)
libboost_thread.so.1.49.0 => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libboost_thread.so.1.49.0 (0x00007f85d2f8a000)
libopenblas.so.0 => /usr/lib/libopenblas.so.0 (0x00007f85d0ef5000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f85d0ccc000)
libxerces-c-3.1.so => /home/ken/bin/BarraOptimizer8.5/lib/intel64/libxerces-c-3.1.so (0x00007f85d07c4000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f85d0499000)
libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007f85d027f000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f85d0040000)

However, with the same source, under Ubuntu 17.10, ldd shows that the shared objects libbarraopt.so links against are not resolved even though -Wl,-rpath is secified: (BARRA_OPS_HOME = /home/renkun/bin/BarraOptimizer8.5)

linux-vdso.so.1 =>  (0x00007ffe067f5000)
libbarraopt.so => /home/renkun/bin/BarraOptimizer8.5/lib/intel64/libbarraopt.so (0x00007f3dc5f0c000)
libR.so => /usr/lib/libR.so (0x00007f3dc58e4000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f3dc555e000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3dc5208000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3dc4ff1000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3dc4c11000)
liboptsrvr.so => not found
libopsproto.so => not found
libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x00007f3dc49b6000)
libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007f3dc4770000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f3dc44fe000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f3dc42d8000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f3dc40c8000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f3dc3eab000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f3dc3ca3000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3dc3a9f000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f3dc3870000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3dc3651000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3dc6526000)
libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007f3dc13ab000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f3dc1182000)
libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x00007f3dc0da3000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f3dc0b63000)

It looks like only libbarraopt.so is linked with the correct path but shared objects it links against are missing.

I'm wondering what might be wrong with my build configurations that breaks under the toolchain shipped by 17.10. Although using global config such as ldconfig would solve such problem, I prefer not because some .so it relies have conflict with the version the OS ships. I'd rather use a locally configured version without affecting the global config.

Driftwood answered 5/11, 2017 at 1:53 Comment(1)
Well just got home ... but let me assure you that just this week I set a new project up with -rpath under 17.04 and it worked as expected... A more minimal example may reduce the barrier to helping you here. – Verlie
B
15

It seems that -Wl,-rpath is not effective here.

What likely happened is that your updated linker emits DT_RUNPATH dynamic tag, where the old linker emitted DT_RPATH. (It's also possible that your old linker was GNU-ld, and the new one is Gold.)

The DT_RUNPATH is preferred as more correct, and affects search path of the binary itself, but not of any of the dependent libraries.

The DT_RPATH has global effect, similar to adding the directory to LD_LIBRARY_PATH environment variable.

You can verify this with: readelf -d a.out | grep 'R.*PATH'.

If you do see the RPATH vs. RUNPATH difference, and in fact are using Gold, you can force the "old" behavior with -Wl,--disable-new-dtags (GNU ld also had --disable-new-dtags added to it recently, so it should work for either linker).

Bluish answered 11/11, 2017 at 22:39 Comment(13)
Yes, RUNPATH instead of RPATH appear in ELF. In my case, RPATH is what I need. – Driftwood
Looks like the default behavior of ld shipped with Ubuntu 16.04 is different from that with Ubuntu 17.10. I look at both locations that /usr/bin/ld points to, they both point to x86_64-linux-gnu-ld but with different version. Is one GNU-ld and the other Gold or just version breaking? – Driftwood
@KunRen I don't have easy access to either 16.04 of 17.10. You can run ld --version. GNU-ld will look like this: GNU ld (GNU Binutils for Ubuntu) .... Gold will look like this: GNU gold (GNU Binutils for Ubuntu ...) – Bluish
Ubuntu 16.04: GNU ld (GNU Binutils for Ubuntu) 2.26.1 Ubuntu 17.10: GNU ld (GNU Binutils for Ubuntu) 2.29.1 – Driftwood
@KunRen It's also possible that GCC invokes something other than /usr/bin/ld (i.e. GCC on 17.10 may be configured to invoke Gold). Use gcc --print-prog-name=ld to find out. – Bluish
@EmployedRussian you closed my question as a dup of this. I don't get any results from readelf for DT_R.*PATH (I do get RUNPATH by itself) – Framing
@SteveLorimer Right. DT_ is not part of the output. Answer corrected. – Bluish
@EmployedRussian thanks! I have tested on both Ubuntu 16.04 and 17.10 and -Wl,--disable-new-dtags flag has fixed my problem. – Framing
@EmployedRussian the comments at the bottom of this blog post mention DT_RPATH is considered a flaw in the ELF specification. How does one propagate the RUNPATH to transitive dependencies? Is the "fix" to set RUNPATH on the dependencies themselves? Is there a better way? TIA. – Framing
@SteveLorimer Yes: the idea is that every ELF image should be built to correctly describe its own required runtime paths and its own dependencies, rather than depending on which "main" application it is loaded into. Setting RUNPATH on each of the transitive dependencies is a way to achieve that. – Bluish
How can I get ld to embed RPATH and not RUNPATH? This would be a quick solution for a similar issue I'm dealing with until I can get around to migrate all my build to the C++ 11 linkage convention and eliminate much of the complexity. – Rockies
@EmployedRussian - I couldn't find any suggestion as to how can I get ld to embed RPATH and not RUNPATH. I might be missing something here, but I'd appreciate a clearer pointer. – Rockies
@YitzikC Use -Wl,--disable-new-dtags flag. – Bluish

© 2022 - 2024 — McMap. All rights reserved.