How to use alternate glibc with existing libstdc++?

Question

I need to use a self-compiled version of glibc (2.18), newer than the default one on the system (2.15). I can compile&link a C++ program, but when I try to run it, I get errors about libstdc++.so.6. (C programs seems to work just fine.) Do I need to recompile gcc against the newer glibc for this to work? Why? (Update: I figured this part out, but I have a few other questions at the bottom.)

Here is a sample C++ program:

#include <iostream>
int main()
{
    std::cout << "ok\n";
    return 0;
}

Following this answer, I compiled it with:

g++ -Wl,--rpath=/path/to/glibc-2.18/lib -Wl,--dynamic-linker=/path/to/glibc-2.18/lib/ld-2.18.so a.cpp

It compiles with no errors, then ldd says:

$ ldd a.out 
    linux-vdso.so.1 =>  (0x00007fff421fe000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f3b96e7f000)
    libc.so.6 => /path/to/glibc-2.18/lib/libc.so.6 (0x00007f3b96ad1000)
    libm.so.6 => /path/to/glibc-2.18/lib/libm.so.6 (0x00007f3b967cf000)
    /path/to/glibc-2.18/lib/ld-2.18.so => /lib64/ld-linux-x86-64.so.2 (0x00007f3b9719d000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3b965b9000)

But when I try to run it:

$ ./a.out 
./a.out: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

This is confusing because it looks like ldd finds libstdc++.so.6 just fine (the specific version is libstdc++.so.6.0.16).

Update: The problem seems to have been (not sure) that the new 2.18 dynamic linker is using its own library path which includes only subfolders of /path/to/glibc-2.18/lib. I got the program to run by adding this new path, followed by the standard paths (/lib', '/usr/lib, etc) to /path/to/glibc-2.18/etc/ld.so.conf and running /path/to/glibc-2.18/sbin/ldconfig. More questions:

1. Do I absolutely need the new 2.18 dynamic linker to run a program with glibc-2.18? Can't the standard linker do it? (This would avoid me having to set up and continuously update the paths of the 2.18 dynamic linker.)

2. If I compile with the 2.18 dynamic linker but without --rpath, the program doesn't work. Why?

3. Should I be using -L/path/to/glibc-2.18/lib in the compilation command (in addition to --rpath and --dynamic-linker)?

Spalding

Seems likely... Maybe strace might give some insight as to whether it's actually reading the file and getting some other error. – Acord

I tried strace, and it shows that the 2.18 dynamic linker is only looking for libstdc++ in subfolders of /path/to/glibc-2.18/lib. I tried without -Wl,--dynamic-linker (so, using the standard one) and then I get SEGV. – Spalding

Can you use LD_LIBRARY_PATH or LD_PRELOAD to coax it into working? – Acord

I managed to make it work by adding /path/to/gdb-2.18/lib followed by the standard paths (/lib, /usr/lib, etc) to /path/to/gdb-2.18/etc/ld.so.conf and running /path/to/gdb-2.18/sbin/ldconfig. My understanding is that this would tell the 2.18 linker to look in the old paths as well. However, now I'm confused as to why I still need --rpath in this case. – Spalding

Answer 1

Do I absolutely need the new 2.18 dynamic linker to run a program with glibc-2.18?

Yes (well, almost. See footnote).

This would avoid me having to set up and continuously update the paths of the 2.18 dynamic linker.

A common technique is to create a g++ shell wrapper, e.g. g++glibc2.18, and encapsulate adding the necessary link arguments there. Then a simple make CXX=g++glibc2.18 would do the right thing.

Can't the standard linker do it?

No. See this answer for explanation.

If I compile with the 2.18 dynamic linker but without --rpath, the program doesn't work. Why?

See the same answer.

Should I be using -L/path/to/glibc-2.18/lib in the compilation command (in addition to --rpath and --dynamic-linker)?

Yes, if you want to use symbols that are present in glibc-2.18 but not present in your system library. Otherwise, no.

Footnote:

As an alternative, you could build your program without the special flags, then use "explicit loader invocation" to actually run it: /path/to/glibc-2.18/lib/ld-2.18.so /path/to/a.out.

Beware: this doesn't always work: if the program likes to re-exec itself (and under other rare conditions). You may also have trouble debugging it when it is invoked that way.

Savino

I'm accepting this, even though I still don't understand number 2, the --rpath issue. If you have a moment, can you please clarify? (I don't see how the linked answer explains it.) – Spalding

Once I (a) I bake in the new rtl (ld-2.18.so), and (b) configure the new rtl's lib path to include the original ones (not covered in the other answer), why do I still need to bake in a run path? The new rtl should know about /path/to/glibc-2.18/lib because that is where it comes from. Is this an issue about directory ordering within the new rtl's lib path?? Because I tried placing /path/to/glibc-2.18/lib as the first element in /path/to/glibc-2.18/etc/ld.so.conf but it still doesn't work (without --rpath). – Spalding

@mmd If /path/to/glibc-2.18/lib is /path/to/glibc-2.18/etc/ld.so.conf, then -rpath should not be necessary. You didn't say how the application fails when you don't have -rpath. You also didn't say how glibc-2.18 was configured. Both may be important for answering your question. – Savino

So: First I followed your previous answer, but the program crashed because it could not find libstdc++ (my original question). Second, with strace I figured ld-2.18.so didn't find libstdc++ because it was only looking in subdirs of /path/to/glibc-2.18/lib. Third, I created /path/to/glibc-2.18/etc/ld.so.conf, put /path/to/glibc-2.18/lib at the top, followed by the std paths (/lib, /usr/lib, etc), and ran /path/to/glibc-2.18/sbin/ldconfig (with -v, confirming that libc.so.6 -> 2.18 is seen before -> 2.15). Now, if I use --rpath it works. [cont] – Spalding

If I don't use --rpath (my question 2), I get a SEGV. strace shows /path/to/glibc-2.18/etc/ld.so.cache being the first successful open() call, so the libpath of ld-2.18.so is being used. But it also shows /lib/x86_64-linux-gnu/libc.so.6 being opened, which is libc-2.15.so, hence the SEGV. So my question 2 becomes, why is libc-2.15.so used by ld-2.18.so even though during its own ldconfig, libc-2.18.so is seen before libc-2.15.so? – Spalding

I asked this (with more details) as a separate question: #20578138 – Spalding