Relative-to-executable path to ld-linux dynamic linker/interpreter

Question

I want to ship and archive binaries (executables with libraries) which are backward and forward compatible with as many Linux distributions as possible and the whole package relocatable. As I understand system libraries like libc also need to be shipped because the executable will crash given a different version of libc. Meanwhile libc seems to be coupled with ld-linux (e.g. binaries compiled on Debian testing already does not work on Ubuntu 18.04 LTS), so I need to package ld-linux too.

My solution is to put all executables and libraries into one directory and set rpath to $ORIGIN (either by linking with -Wl,rpath=$ORIGIN or setting with chrpath or patchelf). This makes the libraries relocatable together with the executable and works for all the libraries except ld-linux which is the linker itself.

It is possible to change the dynamic-linker path by -Wl,--dynamic-linker=/my/path/ld-linux.so or set it with patchelf but the path has to be absolute:

1. The $ORIGIN trick does not work
2. The relative path like ./ works but only when the current directory is the same as the loader itself (the executable crashes with errors when started from elsewhere)
3. I can write a shell script to detect all paths and launch the executable with /my/path/ld-linux.so /my/path/myexecutable $@, but it means yet another layer of indirection and overhead I would like to avoid.

Is there a way to set the path to ld-linux relative to the executable directly into executable?

Maybe there is a way to statically link the ld-linux loader?

Gilding

Struggling with the same problem. And thanks for mentioning your option 2 and 3, I didn't yet thought of that. But yes, forward compatibility with GLIBC is a horrible process, and I have seen too many of SO posts about this. – Oestrogen

musl.libc.org may help you out. It seems a libc implementation with forward compatibility and supports static linking. I have no experience with it, although with my current experiences with GNU libc, I may look into it in the future. It would be nice if we could keep each other informed. – Oestrogen

@Oestrogen currently we use #3 solution, worked fine until recently, where it breaks on some minimalistic Linux distributions, which ship much newer version of libc and newer loads some internet libraries which have to be dynamically linked and conflict with the shipped libc. Then the workaround is to throw away all those wrapper scripts and run the binaries directly. I am thinking to link against some oldest libc I can find in LTS releases, load with whatever libc the host comes with and hope that nobody runs it on outdated distribution (which might still happen on cluster/grid systems) – Gilding

Linking against an old GLIBC with a recent compiler is quite cumbersome, revealed a quick Google search. A special build environment exists to accomplish that, see github.com/phusion/holy-build-box You may also consider a build system using Alpine Linux or Void Linux, both support Musl libc. Disclaimer: I do not (yet) have experience with the just mentioned suggestions. – Oestrogen

@Bart, I have compiled GLIBC, things are much simpler today and it gets easier, just follow their instructions. – Gilding

Good to know, thanks for mentioning. In the future I planned to look (again...) into portable compiling. I will then also take into account compiling glibc2.1 (then symbol versioning was introduced) and let a recent compiler use glibc2.1. Can you recommend a particular tutorial for the last-mentioned approach? – Oestrogen

Answer 1

As I understand system libraries like libc also need to be shipped because the executable will crash given a different version of libc.

That is incorrect: GLIBC guarantees backward compatibility (an executable built on an older system will continue to work on newer versions of GLIBC).

The only sane way to achieve what you want is to compile against the oldest version of GLIBC you wish to support.

Meanwhile libc seems to be coupled with ld-linux

Correct: libc.so.6 and ld-linux are part of GLIBC, must come from the same build, and any mismatch can lead to catastrophic failure (SIGSEGV inside libc.so.6, or inside ld-linux).

I need to package ld-linux too.

That is complicated: the absolute path to ld-linux is hard-coded into the a.out, and can't be changed. Making a relocatable a.out that can tolerate changes to the path to ld-linux is impossible (short of explicit loader invocation you've already tried; which doesn't work well for executables that re-exec themselves).

Update:

I could try building on an old Ubuntu LTS and get most of backward compatibility, but then I would not get the new C++17 compilers, which basically defeats the whole point of modern software engineering.

You could install newer compiler on the older system, and get C++17 with older GLIBC.

One difficulty with that is that you may require a newer libstdc++.so.6.

The good news is that -Wl,-rpath=$ORIGIN works fine -- it's only GLIBC that is hard to relocate. You could also link the executable against libstdc++.a with --static-libstdc++.

However, there might be licensing implications in doing either (but then again your plan already included distributing all the libraries, so that issue is not new).

Plywood

Thanks for looking at it. I agree that GLIBC puts lots of effort into backward compatibility, but I am having issues with forward compatibility: Debian-compiled stuff does not work on recent Ubuntu. They are very close and I did not even get to Fedora and others. I could try building on an old Ubuntu LTS and get most of backward compatibility, but then I would not get the new C++17 compilers, which basically defeats the whole point of modern software engineering. – Gilding

I cannot link libc and libstdc++ statically because other (LGPL) libraries need to link back to them (maybe -rdynamic would solve), and yes, static linking everything would break the LGPL. Therefore the project has to be linked dynamically, and then we are back at the libc linker. – Gilding

@Gilding "and then we are back at the libc linker" -- we are not. Link against old GLIBC and new libstdc++ (dynamically) -- that is the only approach that may work. Well, you could also require your customers to set up a chroot environment, or a container, but these have their own drawbacks. – Plywood