Why does g++ detect undefined reference when dynamically linking

Question

I am probably mistaken about how dynamic linking works, because I cannot figure this out. As I understood it, when a library is dynamically linked, its symbols are resolved at runtime. From this answer:

When you link dynamically, a pointer to the file being linked in (the file name of the file, for example) is included in the executable and the contents of said file are not included at link time. It's only when you later run the executable that these dynamically linked files are bought in and they're only bought into the in-memory copy of the executable, not the one on disk.

[...]

In the dynamic case, the main program is linked with the C runtime import library (something which declares what's in the dynamic library but doesn't actually define it). This allows the linker to link even though the actual code is missing.

Then, at runtime, the operating system loader does a late linking of the main program with the C runtime DLL (dynamic link library or shared library or other nomenclature).

I am confused as to why g++ seems to expect the shared object to be there when dynamically linking against it. Sure, I would expect the name of the library to be necessary so that it can be loaded at runtime, but why is it the .so necessary at this stage? Furthermore, g++ complains about undefined references when linking against the library.

My questions are:

1. Why does g++ seem to require the shared object when dynamically linking against it if the loading of the library only happens at runtime? I understand how the -l flag could be necessary to specify the name of the shared object so that it can be loaded in runtime, but I see no point in having to provide the path to the .so at link time (-L) or the .so itself.
2. Why does g++ attempt to resolve the symbols when dynamically linking? Nothing stops me from having a complete .so at link time but then providing a different (incomplete) .so at runtime, which causes the program to crash when it tries to use an undefined symbol.

I made a reproducible example:

Directory structure:

.
├── main.cpp
└── test
    ├── usertest.cpp
    └── usertest.h

---

File contents:

test/usertest.h

#ifndef USERTEST_H_4AD3C656_8109_11E8_BED5_5BE6E678B346
#define USERTEST_H_4AD3C656_8109_11E8_BED5_5BE6E678B346

namespace usertest
{
    void helloWorld();

    // This method is not defined anywhere
    void byeWorld();
};

#endif /* USERTEST_H_4AD3C656_8109_11E8_BED5_5BE6E678B346 */

test/usertest.cpp

#include "usertest.h"
#include <iostream>

void usertest::helloWorld()
{
    std::cout << "Hello, world\n";
}

main.cpp

#include "test/usertest.h"

int main()
{
    usertest::helloWorld();
    usertest::byeWorld();
}

---

Usage

$ cd test
$ g++ -c -fPIC usertest.cpp
$ g++ usertest.o -shared -o libusertest.so
$ cd ..
$ g++ main.cpp -L test/ -lusertest
$ LD_LIBRARY_PATH="test" ./a.out

Expected behaviour

I would expect everything to crash when attempting to launch a.out because it cannot find the necessary symbols in libusertest.so.

Actual behaviour

The building of a.out fails at link time because it cannot find byeWorld():

/tmp/ccVNcRRY.o: In function `main':
main.cpp:(.text+0xa): undefined reference to `usertest::byeWorld()'
collect2: error: ld returned 1 exit status

Gulick

I think it needs the shared lib to get the header file, else the functions from your .so are unknown. But I'm not sure. – Merridie

Possible duplicate: #48771458 – Merridie

@rustyx Isn't that a (non-)issue at link time too? I can't see the connection to my question, sorry – Gulick

@Merridie The header files have to be provided independently from the .so, and g++ gets ahold of them via the -I flag, so it can't be that :/ Also, headers are involved at compile time, not link time. – Gulick

Answer 1

With the ELF format it indeed isn't necessary to know which symbols belong to which library, as the actual symbol resolution happens when the program is executed. By convention though ld will still resolve the symbols when producing the binary. It's for your convenience, so that you get immediate feedback when you have missing symbols, since in that case the chance is big your program won't work.

Using the --warn-unresolved-symbols flag you can change ld behavior in this case from an error to a warning:

$ g++ -Wl,--warn-unresolved-symbols main.cpp -lusertest

Should emit a warning but still create the executable. Note that you still need to provide the library name, otherwise ld won't know where to look for the needed symbols.

On Windows, the linker needs to know exactly which symbol belongs to which library in order to produce the necessary import tables. So it is impossible to build a PE binary with unresolved symbols.

Codification

I did not know the concept of import table. How is this an issue that cannot be deferred to runtime? Wouldn't it make sense to do this when loading the libraries prior to entering main at runtime? – Gulick

@Gulick I think import tables is a windows specific concept, but the commands you provided and library extensions ".so" are linux specific. – Esmeraldaesmerelda

Ah, thanks for the update! This is very interesting: does that mean that @miravalls' answer is wrong in its assumption about the fact that we need to know which symbol belongs to which library by the time we link the executable? Regardless of whether we get a warning or an error, do the symbols get resolved at runtime? – Gulick

@Codification +1. I find it amusing that you can create a broken executable that may be fixed by updating a library. – Esmeraldaesmerelda

An ELF binary contains a list of dependent modules (shared libs) and a separate list of "undefined symbols". So essentially linking happens every time the program starts. – Codification

Answer 2

The code segment of an executable is always read-only as a security measure, so you can not have a program that modifies its own code at runtime. As others have mentioned, what the linker is doing is generating a list of what symbols are provided per library.

You suggest this process could be deferred to run time, but that would mean that your binary could crash every time you launch it if the list of libraries you provided at link time was incomplete. Why would you risk that when you can simply check that at link time? Deferring symbol resolution to runtime would mean that each time you run your program it would perform the same search in all its dependencies for all unresolved symbols. Furthermore, if you did not have to give the list of libraries at link time, it would mean that it would have to try all possible libraries at runtime. How would you resolve a symbol that is defined by multiple libraries?

As I understand (in a very simplified way), what the dynamic linker does at runtime is keep a hash table that translates those symbols into addresses (function pointers) in the dynamically linked library after it is mapped in your program's address space. In your executable, the linker needs to know which library provides each symbol (function, variable, etc) to perform this resolution.

So, in this very simplified explanation, your call to usertest::helloWorld(); is translated to something like dynamic_resolve("usertest::helloWorld", "libusertest.so")(); with dynamic_resolve receiving the symbol name and the library name, and returning a function pointer. Internally, what dynamic_resolve (made-up name) is doing is loading the library "libusertest.so", retrieving the address of the function in the library, caching this in a hash table, and then return the function pointer. It is probably using these system calls. After the first call, as the result is cached in a hash table and the library is already loaded, all subsequent calls are much cheaper.

Esmeraldaesmerelda

Thanks for the answer. I understood the necessity to list the name of the libraries (similar to -l flag), just not the necessity for the libraries to be there. If I got you right, it's both an issue of efficiency (not having to resolve what library each symbol belongs to every time), not mixing library versions (even though it's not fool-proof) and technical constrains (needing the name of the library for each symbol for the dl functions). – Gulick

@Gulick Yes, I think so. – Esmeraldaesmerelda