Using C++11 multithreading in shared library loaded by program without thread support
Asked Answered
I

3

18

I am currently trying to use C++11 multithreading in a shared library that is loaded into the main program (written in C) on Linux. This is part of a big simulation program and I cannot change anything about the library loading or change the main program in general.

The main program is compiled with gcc 4.1.2 and I don't have the sources for it (I cannot recompile it with gcc 4.8.2).

The shared library is compiled with gcc 4.8.2 in order to use C++11 multithreading. I am passing the compiler commands

-pthread -lpthread -std=c++11

as explained in What is the correct link options to use std::thread in GCC under linux?

Compiling a standalone test program with this configuration ("-pthread -std=c++11" and gcc 4.8) works correctly on my system. But when I start the program loading the shared library I get an exception:

Caught std::exception!
Exception Message: Enable multithreading to use std::thread: Operation not permitted

Terminating...

Using the -pthread and -lpthread (Edit: and also only -pthread without -lpthread) compile parameter doesn't work. The compiler arguments are (I am using the cook build system):

-pthread -std=c++11 -fmessage-length=0 -fPIC -Wchar-subscripts ...(lots of -W* here)
... -Wunused-variable -m64 -D__64BIT__ -pthread -lpthread

and the linker arguments (duplicate parameters due to the build system):

-pthread -lpthread -std=c++11 -pthread -lpthread -std=c++11 -shared -fPIC -Wl,-Bsymbolic -Wl,--allow-shlib-undefined -pthread -lpthread

calling ldd on my library gives the following output

$ ldd calc3/build/amd64_linux26_RH5/library.so
    linux-vdso.so.1 =>  (0x00007fff4d1fd000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00002ae6ec124000)
    libstdc++.so.6 => /afs/bb/data/d6833/util/gcc_482/lib64/libstdc++.so.6 (0x00002ae6ec340000)
    libm.so.6 => /lib64/libm.so.6 (0x00002ae6ec655000)
    libgcc_s.so.1 => /afs/bb/data/d6833/util/gcc_482/lib64/libgcc_s.so.1 (0x00002ae6ec8d8000)
    libc.so.6 => /lib64/libc.so.6 (0x00002ae6ecaef000)
    /lib64/ld-linux-x86-64.so.2 (0x00000032cb400000)

and on the main program

$ ldd .../bin-64/main_program
    linux-vdso.so.1 =>  (0x00007fff64595000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00000032cc000000)
    libz.so.1 => /usr/lib64/libz.so.1 (0x00000032cc800000)
    libc.so.6 => /lib64/libc.so.6 (0x00000032cb800000)
    /lib64/ld-linux-x86-64.so.2 (0x00000032cb400000)

The pthread library is linked to my shared library but not to the main program. This answer states that you have to link pthreads to the main program but the 2nd comment on this answer (by @R..) says it isn't necessary (which sounds logically).

Unfortunately I don't know anything about the loading mechanics of the whole system, except that my library is using another C++ library as API.

Note that other C++11 features do work (and libstdc++.so is in the dependencies of my library) but C++11 multithreading is not (although libpthread.so is also in the dependencies of my library).

Using a threading class from a library contained in the program itself is working (and this thread class seems to use pthreads too).

I have also tried to use -fabi-version=0 or -fabi-version=2 because the main program is compiled with gcc 4.1.2 with my library but it didn't change anything.

Is there anything I have overlooked or a compiler option I can use to make it work? Or does it seem to be a problem of my program environment? Any ideas are welcome.

Edit:

I tried using -Wl,-no-as-needed (as suggested in the comments) but it unfortunately didn't change anything.

Using clang 3.5 instead of gcc 4.8 also didn't work.

Creating a small test application which loads a shared library (like in the answer below by @chill) works (even without the compiler flag) as long as I use gcc 4.8 or clang 3.5 for both main application and shared library. When using gcc 4.1 for the main program however the main program even fails to load the library (which is working in my 'real' application). I think there might be a problem with the differen ABIs of the compilers.

Using pthreads directly from pthread.h seems to work (although the program currently terminates on pthread_join without error message, but I'm still testing there...)

Edit 2:

Running the 'test program' with LD_LIBRARY_PATH=$(pwd):$LD_LIBRARY_PATH (because the gcc 4.8 library paths need to be in there too, thanks @MvG) did run the program but crashed again with the Enable multithreading to use std::thread: Operation not permitted exception.

I checked all other libraries that are loaded (found them with strace ./main_program 2>&1 | grep '^open(".*\.so"' [see here]) and checked them all with ldd. They all depend on the same libraries (with the same paths). ldd outputs (on all of them):

linux-vdso.so.1 =>  (0x00007fff4d3fd000)
libstdc++.so.6 => /afs/bb/data/d6833/util/gcc_482/lib64/libstdc++.so.6 (0x00002ade28774000)
libm.so.6 => /lib64/libm.so.6 (0x00002ade28ab0000)
libgcc_s.so.1 => /afs/bb/data/d6833/util/gcc_482/lib64/libgcc_s.so.1 (0x00002ade28d33000)
libc.so.6 => /lib64/libc.so.6 (0x00002ade28f49000)
/lib64/ld-linux-x86-64.so.2 (0x00000032ea200000)

(that they all don't depend on libpthread.so.0 except my library and one other (but it's the same /lib64/libpthread.so.0))

Some libraries do have more dependencies (which don't seem thread-related) but there don't seem to be any 'conflicting' dependencies (there are no dependencies to different versions/paths of the same library in any of these libraries).

Idle answered 13/12, 2013 at 13:54 Comment(6)
#19464102 May be this can help.Sprang
Have you tried using just '-pthread'? I don't think you need both -lpthread and -pthread.Chavis
Have you seen : https://mcmap.net/q/223069/-compiling-multithread-code-with-g ? The accepted answer says it's a gcc bug and provides workaround optionsBrochure
Those duplicated linker options are ugly. I'd assume you'll also have trouble getting the --no-as-option injected in the right spot. It needs to go before the -lpthread option.Heartwarming
Just guessing, but could it be that some other library already caused loading and initialization of libstdc++ in a situation where threading was not available?Mordvin
I've been able to reproduce your problem. In my setup, the order of libraries is really important: if a library using libgcc gets loaded before the first library using libpthread, then you have no C++ thread support. This is due to how __gthread_active_p and weak symbol resolution works: the library resolves the symbol using all libraries loaded before it was loaded. Now all we need is a work around this problem.Mordvin
M
12

In thread.cc you can read that this exception is generated if __gthread_active_p returns false. That call simply checks whether or not a given symbol is available. The symbol in question is a weak symbol: it does not have to be present, but its presence is checked to decide whether or not threading is supported.

But what does presence of a symbol mean? In this case it means that the symbol is in the list of symbol tables which the library in question (libgcc_s.so.1 in my case) searches for symbol definitions. That includes symbols exported by the application itself, but also symbols exported by all the libraries loaded before it. However, it does not include libraries loaded afterwards. Unfortunately, if libgcc is loaded before libpthread, then the symbol is not available on its search domain. So it reports threading as unsupported. I guess you have some other C++ module loaded before the multi-threaded one, so you encounter this problem.

One solution which works in my reproducing example is setting LD_PRELOAD=/lib64/libpthread.so.0 in the environment used to call the binary. That loads libpthread up front, so its symbols are available to satisfy the weak symbol linkage. This won't work for setuid/setgid binaries, and might be considered an ugly hack in other cases as well, so I'm interested in cleaner solutions. Nevertheless, this gets the job done most of the time.

Mordvin answered 16/12, 2013 at 16:44 Comment(2)
Setting LD_PRELOAD=/lib64/libpthread.so.0 was the solution, thanks! It might not be the prettiest solution but it works. And good explanation of the library symbol / loading mechanics!Idle
thanks for helping FinFET (who is actually my coworker). But as FinFET had not enough reputation I gave him some of mine for bumping this problem with a Bountyhunt.Goldfish
B
3

I have been able to reproduce something very similar to your situation.

First the test shared library source:

#include <thread>

void f() {}

extern "C" int foo () {
        std::thread(f).join();
        return 0;
}

Compiled/linked with c++ -fPIC -shared -std=c++11 -pthread thrlib.cxx -o libthrlib.so

Then the "main program":

#include <dlfcn.h>

int
main () {
    void *lib = dlopen ("libthrlib.so", RTLD_NOW);

    int (*foo)();
    foo = (int (*)()) dlsym (lib, "foo");
    foo ();
}

Compiled/linked with gcc main.c -ldl

The result of execution is:

velco@sue:~/tmp$ LD_LIBRARY_PATH=`pwd` ./a.out 
terminate called after throwing an instance of 'std::system_error'
  what():  Enable multithreading to use std::thread: Operation not permitted
Aborted (core dumped)

Now, build the shared library, by adding to the command line options -Wl,-no-as-needed

c++ -fPIC -shared -std=c++11 -pthread thrlib.cxx -o libthrlib.so -Wl,-no-as-needed

And the main program executes without an error.

I run this on Ubuntu 13.10 with gcc 4.8.1, all of the above may in fact not be applicable to your environment, but the effects are very similar to yours, so there's a good chance the same solution will work for you, worth a try anyway :)

PS. Another thing worth trying, add an explicit hard reference to pthread_cancel in your library:

#include <pthread.h>
void *zzz = (void *) pthread_cancel;
Bastia answered 16/12, 2013 at 10:18 Comment(5)
So this answer is the same as provided in the answer linked to in comment: https://mcmap.net/q/223069/-compiling-multithread-code-with-g (but still a good write-up)Ary
Unfortunately using -Wl,-no-as-needed doesn't seem to fix the problem. Your test program even runs without it. However if I use gcc 4.1 for the main program the library even fails to load at all.Idle
I can't reproduce the error, using gcc 4.1.2 for main and g++ 4.8.2 for the library. I modified main to check for errors and print dlerror() output if dlopen or dlsym return NULL. I also modified the path to "./libthrlib.so" so I don't have to fool around with LD_LIBRARY_PATH. After this, the program executes even if compiled with -Wl,--as-needed. And since OP indicates pthread in its ldd output, I doubt issues due to --as-needed.Mordvin
@FinFET, did you have a chance to try that pthread_cancel thing?Bastia
@Mordvin I got something to work with LD_LIBRARY_PATH, see Edit 2 in my question. @Bastia I did try ... (void*)pthread_cancel; too, did not change anything unfortunately.Idle
K
0

LD_PRELOAD=/lib64/libpthread.so.0 should resolve the issues of missing __pthread_key_create symbol when pthread is not loaded.

But preloading the whole pthread object might break other features when people are trying to write a shim or wrapper for pthread.

So a better way is to compile a simpler shared object with only this symbol and preload it. For example:

// pthread_patch.c
// don't need to include any header
int
__pthread_key_create (unsigned int *key, void (*destr) (void *))
{
    return 1;  // EPERM
};

Compile with: gcc -shared -o pthread_patch.so pthread_patch.c -fPIC

And then run application with LD_PRELOAD=pthread_patch.so.

If pthread_cancel is missing, you could add that symbol in the same way.

Kress answered 14/11, 2023 at 17:25 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.