C/C++ How Does Dynamic Linking Work On Different Platforms?

Question

How does dynamic linking work generally?

On Windows (LoadLibrary), you need a .dll to call at runtime, but at link time, you need to provide a corresponding .lib file or the program won't link... What does the .lib file contain? A description of the .dll methods? Isn't that what the headers contain?

Relatedly, on *nix, you don't need a lib file... How how does the compiler know that the methods described in the header will be available at runtime?

As a newbie, when you think about either one of the two schemes, then the other, neither of them make sense...

Mopboard

No, if you load a DLL with LoadLibrary you should not need to link to any specific library. That's the point of dynamically loading libraries. It's the same on other platforms. – Terceira

@JoachimPileborg true, but the OP is (understandably) puzzled about why dynamic libs on Windows have a .lib file associated with the .dll, when it isn't necessary on other platforms. – Angeloangelology

I think but do not know that a Windows .LIB file contains stub definitions which the dynamic linker then replaces. But this is not necessary. If a system uses stub functions, they can be emitted into the executable at compile time. Alternatively, the executable can simply have missing symbols (just like an object file) and the dynamic linker really goes and links them at startup time just like a static linker would do. I think both of those approaches have been used on unix. – Churlish

@Jalf - This is exactly what I'm talking about. And when you think about it, it does, in some way make sense... The program should know what methods are due to be available. But on *nix, I think the idea is that the header files and/or member declarations provide the compiler with the info it will need at runtime... Like I say, for a newbie (and a few vets, I expect), it's quite a large, and confusing distinction. – Mopboard

Programming languages try to help you find bugs at compile time. That is one reason they have types and the linker helps find bugs by checking everything you call is defined. When you use LoadLibrary() you are bypassing all of these checks. It helps to have an assembly language background. – Incognizant

Not sure where most people are getting their windows information from on here, but you certainly DO NOT need a .lib file to use LoadLibrary or link its .lib file, nor to call its functions this is completely wrong. – Myrta

@Myrta - From what I've managed to glean so far, no, you don't need a .lib file to link your Windows program, but you'll have to write a fair amount of confusing inline boilerplate code on Microsoft platforms to avoid using an import library .lib file. On *nix, you need neither an import library, or extraneous boilerplate code in the most common form of dynamic loading (with dlopen). Apparently, Microsoft used to do it in exactly the same way as on *nix. I'm wondering why that changed. – Mopboard

From what I can tell dlopen and dlsym are the same as LoadLibrary/GetProcAddress so the boiler plate code is actually the same? – Myrta

@Mopboard As far as I know, if you just call dlopen without dlsyms following that call, you'll get nothing. So Windows and *nix are similar when it's about dynamic DLL loading. The are different when it's about linking to the dll at link-time. – Markley

Answer 1

To answer your questions one by one:

• Dynamic linking defers part of the linking process to runtime. It can be used in two ways: implicitly and explicitly. Implicitly, the static linker will insert information into the executable which will cause the library to load and resolve the necessary symbols. Explicitly, you must call LoadLibrary or dlopen manually, and then GetProcAddress/dlsym for each symbol you need to use. Implicit loading is used for things like the system library, where the implementation will depend on the version of the system, but the interface is guaranteed. Explicit loading is used for things like plug-ins, where the library to be loaded will be determined at runtime.

• The .lib file is only necessary for implicit loading. It contains the information that the library actually provides this symbol, so the linker won't complain that the symbol is undefined, and it tells the linker in what library the symbols are located, so it can insert the necessary information to cause this library to automatically be loaded. All the header files tell the compiler is that the symbols will exist, somewhere; the linker needs the .lib to know where.

• Under Unix, all of the information is extracted from the .so. Why Windows requires two separate files, rather than putting all of the information in one file, I don't know; it's actually duplicating most of the information, since the information needed in the .lib is also needed in the .dll. (Perhaps licensing issues. You can distribute your program with the .dll, but no one can link against the libraries unless they have a .lib.)

The main thing to retain is that if you want implicit loading, you have to provide the linker with the appropriate information, either with a .lib or a .so file, so that it can insert that information into the executable. And that if you want explicit loading, you can't refer to any of the symbols in the library directly; you have to call GetProcAddress/dlsym to get their addresses yourself (and do some funny casting to use them).

Saltatorial

With two separate files, you can link against a DLL without actually having the DLL. That might not be the actual reason they use two files, but it's a plausible reason. – Tetratomic

The question is about dynamic linking but it may be good to know that .lib files can also be used for static linking, essentially linking the libraries content into the executable, making DLLs unnecessary. – Salzburg

Re point 2: You could point out that this problem is MSVC-specific, ant it's possible to link to the DLL itself on MS Windows platform. See edit to my answer. – Markley

I guess it is worth to say, that the .lib file is not required to use the DLL. It just simplifies the process. – Cingulum

@Salzburg It's not the same .lib. Microsoft uses the same extension for two different types of files. – Saltatorial

@JamesKanze Wow, didn't know that. Although it makes sense because a regular .lib file doesn't need to contain any compiled code. – Salzburg

Answer 2

The .lib file on Windows is not required for loading a dynamic library, it merely offers a convenient way of doing so.

In principle, you can use LoadLibrary for loading the dll and then use GetProcAddress for accessing functions provided by that dll. The compilation of the enclosing program does not need to access the dll in that case, it is only needed at runtime (ie. when LoadLibrary actually executes). MSDN has a code example.

The disadvantage here is that you need to manually write code for loading the functions from the dll. In case you compiled the dll yourself in the first place, this code simply duplicates knowledge that the compiler could have extracted from the dll source code automatically (like the names and signatures of exported functions).

This is what the .lib file does: It contains the GetProcAddress calls for the Dlls exported functions, generated by the compiler so you don't have to worry about it. In Windows terms, this is called Load-Time Dynamic Linking, since the Dll is loaded automatically by the code from the .lib file when your enclosing program is loaded (as opposed to the manual approach, referred to as run-time dynamic linking).

Narbonne

The .lib file probably doesn't have any information that isn't present (perhaps in another format) in the .dll. I'm not sure why Microsoft decided to use two files (since the existing practice when Microsoft implemented DLLs was to use a single file); I can only guess that it was to provide a means by which you could distribute the libraries necessary to run your executable without distributing the means of creating a program which would link against those libraries. – Saltatorial

@JamesKanze I'm not sure that's entirely true. Dll allows limited reflection, for instance it is possible to enumerate the names of the exported functions. However I'm not sure if it's possible to reconstruct the full function signature from just the dll. – Narbonne

@JamesKanze See this question for discussion on the reflection problem. Seems that it is indeed not possible to reconstruct function signatures from just the dll. – Narbonne

The .dll must contain enough information for the run-time loader to be able to link with it. And the linker doesn't need any more information. (Neither the .dll nor the .lib contain the full function signature, per se. Both do contain the mangled name of the function.) – Saltatorial

And the article you quote doesn't address the issue. The .lib doesn't need any information which isn't in the .dll. – Saltatorial

@JamesKanze You are of course right, the signatures come from the headers and not the .lib files. So it seems that the convenience gain provided by the .lib files for loading could have been just as easily achieved with a smarter runtime loader. – Narbonne

The .lib will hard code the dependency, if foo.dll isn't found your app won't launch, if you call LoadLibrary you can handle this failure – Myrta

I've seen DLLs with the function names stripped, where the functions were only accessible via ordinal. An import library would be able to give names to those functions without exposing them in the DLL. – Upperclassman

@Upperclassman - So perhaps Microsoft's scheme DOES have it's roots in security and ip, not type safety... – Mopboard

@Charlie: It's probably at least partly an IP thing. But keep in mind, too, the Windows ecosystem is quite different from *nix; there's no One True Compiler and no One True Calling Convention in Windows. (There are at least two: cdecl, and stdcall. And any compiler could come up with its own as well.) Having the import library distinct from the DLL allows the differences to be hidden away behind a wrapper function. So the code can use the compiler's native calling convention, and the import library would do the translation. – Upperclassman

Answer 3

How does dynamic linking work generally?

The dynamic link library (aka shared object) file contains machine code instructions and data, along with a table of metadata saying which offsets in that code/data relate to which "symbols", the type of the symbol (e.g. function vs data), the number of bytes or words in the data, and a few other things. Different OS will tend to have different shared object file formats, and indeed the same OS may support several, but that's the gist of it.

So, imagine the shared library's a big chunk of bytes with an index like this:

SYMBOL       ADDRESS        TYPE        SIZE
my_function  1000           function    2893
my_number    4800           variable    4

In general, the exact type of the symbols need not be captured in the metadata table - it's expected that declarations in the library's header files contain all the missing information. C++ is a bit special - compared to say C - because overloading can mean there are several functions with the same name, and namespaces allow for further symbols that would otherwise be ambiguously named - for that reason name mangling is typically used to concatenate some representation of the namespace and function arguments to the function name, forming something that can be unique in the library object file.

A program wanting to use the shared object can generally do one of two things:

• have the OS load both itself and the shared object around the same time (before executing main()), with the OS Loader responsible for finding the symbols and examining metadata in the program file image about the use of those symbols, then patching in symbol addresses in the memory the program uses, such that the program can then just run and work functionally as if it'd known about the symbol addresses when it was first compiled (but perhaps a little slower)

• or, explicitly in its own source code call dlopen sometime after main runs, then use dlsym or similar to get the symbol addresses, save them into (function/data) pointers based on the programmer's knowledge of the expected data types, then call them explicitly using the pointers.

On Windows (LoadLibrary), you need a .dll to call at runtime, but at link time, you need to provide a corresponding .lib file or the program won't link...

That doesn't sound right. Should be one or the other I'd think.

Wtf does the .lib file contain? A description of the .dll methods? Isn't that what the headers contain?

A lib file is - at this level of description - pretty much the same as a shared object file... the main difference is that the compiler's finding the symbol addresses before the program's shipped and run.

Reena

Just a guess, but you come from a Unix background. Under Windows, a .lib can be either a static library (in which case, there is no .dll), or a collection of some of the information from the .dll, used by the linker to generate the information necessary for automaticl library loading. (Under Unix, the linker extracts this information directly from the .so file.) – Saltatorial

My initial reaction when seeing it was that it was stupid, not weird. I've been working on Windows now for over four years, and I've gradually come to believe that there is usually a reason behind their apparent stupidities---a reason usually linked to money, in some way or another. (Although sometimes...) – Saltatorial

As others have noted, if you use LoadLibrary, you don't need the corresponding .lib file. I once used LoadLibrary precisely because my development system did not have the .lib even though the target system had the .dll. – Churlish

@AdrianRatnapala: If you can use LoadLibrary, and find the functions by name to call them, and they actually work, then there's enough information in the DLL to automagically build an import library. Your IDE should include a tool that does just that. VS does, and Borland did back when i used it. – Upperclassman

@Upperclassman I was working in a constrained environment where I couldn't just install VS. I was working with MinGW and a text editor. – Churlish

Answer 4

Relatedly, on OS X (and I assume *nix... dlopen), you don't need a lib file... How how does the compiler know that the methods described in the header will be available at runtime?

Compilers or linkers do not need such information. You, the programmer, need to handle the situation that the shared libraries you try to open by dlopen() may not exist.

Nestorius

That confirms how it works on GCC style compilers, (almost) certainly, but I'm looking for an answer that addresses the similarities and differences across different platforms, and why they exist. – Mopboard

@Mopboard Why they exist is a real question. The Unix .so paradigm was established by Sun, back around 1980. It was never really secret, so Microsoft could have used it when they implemented dynamic loading, some 20 years later. I'm just guessing, but I suspect that it was licensing issues which were at stake (but that would be better solved by linking statically). – Saltatorial

Answer 5

You can use a DLL file in Windows in two ways: Either you link with it, and you're done, nothing more to do. Or you load it dynamically during run-time.

If you link with it, then the DLL library file is used. The link-library contains information that the linker uses to actually know which DLL to load and where in the DLL functions are, so it can call them. When your program is loaded, the operating system also loads the DLL for you, basically what is does it call LoadLibrary for you.

In other operating systems (like OS X and Linux) it works in a similar way. The difference is that on these systems the linker can look directly at the dynamic library (the .so/.dynlib file) and figure out what's needed without a separate static library like on Windows.

To load a library dynamically, you don't need to link with anything related to the library you want to load.

Terceira

I guess that the main point of the question was: Why Windows does need a .lib and Linux/OSX do not? – Markley

Answer 6

Modern *nix systems derive process of dynamic linking from Solaris OS. Linux, particularly, doesn't need separate .lib file because all external dependencies are contained in ELF format. .interp section of ELF file indicates that there are external symbols inside this executable that needed to be resolved dynamically. This comes for dynamic linking.

There is a way to handle dynamic linking in user space. This method is called dynamic loading. This is when you are using system calls to get function pointers to methods from external *.so.

More information can be found from this article http://www.ibm.com/developerworks/library/l-dynamic-libraries/.

Idiographic

On *nix/GCC, at what point would the linker/compiler know it has to account for a dynamic library? As soon as it parses the 'dlopen()' symbol? Or does the linker/compiler pick up on any '#include <dlfcn.h>' directive? Like I say, when you compare one DLL model to the other, they both seem a little strange when juxtaposed... I'll check out the link though, thanks! – Mopboard

You still don't get the point, Charlie. There are two totally different ways of making DLL work: a) Link to it (example: pass GCC -lSDL to link to SDL dynamic library) b) Load it at runtime (example: put this code into your project: void* lib=dlopen("./libSDL.so", RTLD_LAZY); //now import function pointers via dlsym...). In a), when libSDL.so is not available at link time on developer machine, you'll get the link error. In b) you will only get a runtime error when SDL is not available in the current directory of the end-user. – Markley

Answer 7

Like others already said: what is included in a .lib file on Windows is included directly in the .so/.dynlib on Linux/OS X. But the main question is... why? Isn't *nix solution better? I think it is, but the .lib has one advantage. The developer linking to the DLL doesn't actually need to have access to the DLL file itself.

Does a scenario like that happen often in the real world? Is it worth the effort of maintaining two files per DLL file? I don't know.

Edit: Ok, guys let's make things even more confusing! You can link directly to a DLL on Windows, using MinGW. So the whole import library problem is not directly related to Windows itself. Taken from sampleDLL article from MinGW wiki:

The import library created by the "--out-implib" linker option is required iff (==if and only if) the DLL shall be interfaced from some C/C++ compiler other than the MinGW toolchain. The MinGW toolchain is perfectly happy to directly link against the created DLL. More details can be found in the ld.exe info files that are part of the binutils package (which is a part of the toolchain).

Markley

Re the advantage: I suspect that the real advantage is the opposite: you can't link against a .dll, even if you have it (because it was distributed with a program that needed it). There may be a slight advantage when linking, because the .lib will typically be significantly smaller than the .dll. (You still need the .dll when developing against it, since you have to run the program you just linked, in order to test it.) – Saltatorial

@JamesKanze "I suspect that the real advantage is the opposite: you can't link against a .dll, even if you have it" Why is it an advantage? – Markley

The *nix way isn't really "better"; it's barely even different, from the outside. .a files serve most of the same purposes as .lib in Windows. And when you say -lstuff, the compiler looks for a libstuff.a, not a shared library. – Upperclassman

Ah... That's another point of confusion. on Windows platforms .lib files appear to serve a dual purpose (they can either be static or import). I was confused by import .lib's), while on *nix, I think an .a file is just a compressed/compiled collection of .c sources. – Mopboard

@Charlie: An import library is effectively just a static library that knows how to load the DLL, and has stub functions that call into the DLL. And an .a file can contain anything, really (the "a" stands for "archive"), but usually contains .o files. – Upperclassman

@Markley The windows model allows distribution of any needed dynamic objects without allowing others to link against them. (At least, that's what I suspect is the perceived advantage. In most cases, statically linking would be an even better solution, so you wouldn't have to distribute the .dll at all.) – Saltatorial

@Upperclassman When you say -lstuff, the linker looks in the directories specified by -L options, and uses the first it finds of libstuff.a and libstuff.so; if it finds both in the same directory, it will use libstuff.so. (At least, this was true of the Solaris linkers, but I think it's pretty general.) You can modify this behavior with linker options.) – Saltatorial

@JamesKanze: If that's the case, then the GCC docs need updating. They don't mention .so files. The info and man pages describing -l say, "The only difference between using an -l option and specifying a file name is that -l surrounds library with lib and .a and searches several directories." – Upperclassman

@JamesKanze: The man page for ld does say that -l looks for a shared library first, on systems that support them. So either GCC is doing its own thing with -l, or its documentation is incorrect. – Upperclassman

@Upperclassman the *nix way really is "better" and JamesKanze explains why: on Windows you can end up with the .DLLs that you can't link against. There is not much difference, since on Unix you can end up with a .so without the appropriate headers, which is nearly as bad. – Churlish

@AdrianRatnapala: Sometimes, you want DLLs that can't be arbitrarily linked against. Windows isn't *nix, and it (and the people writing code for it) often have differing philosophies. Not everything is free/open, not everything works the same. Nor should it be forced to. Windows gives you more of an option. – Upperclassman

@Upperclassman If you really want to "be a bad guy" and use some DLL without a license to use it, you still can use it via LoadLibrary. It's just not that comfortable. So this whole advantage doesn't speak to me. – Markley

@cubuspl42: Sure, you can use it, if you know what the functions do. But that gets a whole lot harder -- and legally hairier -- with a stripped DLL. "DoSomething" becomes "function #17", and you'd have to reverse-engineer the DLL (or the import library, or an app that uses it) in order to know which functions to call. That puts you further over the line into willful infringement, should the owner of the DLL decide to sue. – Upperclassman

@Upperclassman Well, gcc only passes the parameter on to the linker, so I suppose that it depends on what the linker does. My description is based on experience under Solaris (so Sun's linker). (I suppose that it is very difficult for gcc to document things they don't control. Still, I suspect that the behavior I describe is widespread.) – Saltatorial

@Upperclassman "Windows gives you more of an option": I'm not sure about that. As Adrian points out, without the headers, you really can't do much either. On the other hand, given that both static and dynamic linking require a .lib at build time, it's very difficult to provide both static and dynamic versions of a library. I'd say that there is a slight advantage to the Unix system, even though I think I understand where the Windows system is coming from. (It would have been a lot better had they chosen a different name for the second file, say .sym.) – Saltatorial

@JamesKanze: Headers are just a list of prototypes and structure definitions. Those can easily be deduced (or outright copied) from a tutorial or manual site. (Just as an example, MSDN shows prototypes for every C function in its APIs, and definitions for every structure an apps programmer will need to use.) But that is only so helpful without a way to map function names to addresses. – Upperclassman

Answer 8

Linux also requires to link, but instead against a .Lib library it needs to link to the dynamic linker /lib/ld-linux.so.2, but this usually happens behind the scenes when using GCC (however if using an assembler you do need to specify it manually).

Both approaches, either the Windows .LIB approach or the Linux dynamic linker linking approach, are considered in reality as static linking. There is, however, a difference that in Windows part of the work is done at link time although it still has work at load time (I am not sure, but I think that the .LIB file is merely for the linker to know the physical library name, the symbols however are only resolved at load time), while in Linux everything besides linking to the dynamic linker happen at load time.

Dynamic linking is in general referring to open manually the DLL file at runtime (such as using LoadLinrary()), in which case the burden is entirely on the programmer.

Persian

Answer 9

In shared library, such as .dll .dylib and .so, there is some information about symbol's name and address, like this:

------------------------------------
| symbol's name | symbol's address |
|----------------------------------|
| Foo           | 0x12341234       |
| Bar           | 0xabcdabcd       |
------------------------------------

And the load function, such as LoadLibrary and dlopen, loads shared library and make it available to use.

GetProcAddress and dlsym find you symbol's address. For example:

HMODULE shared_lib = LoadLibrary("asdf.dll");
void *symbol = GetProcAddress("Foo");
// symbol is 0x12341234

---

In windows, there is .lib file to use .dll. When you link to this .lib file, you don't need to call LoadLibrary and GetProcAddress, and just use shared library's function as if they're "normal" functions. How can it work?

In fact, the .lib contains an import information. It's like that:

void *Foo; // please put the address of Foo there
void *Bar; // please put the address of Bar there

When the operating system loads your program (strictly speaking, your module), operating system performs LoadLibrary and GetProcAddress automatically.

And if you write code such as Foo();, compiler convert it into (*Foo)(); automatically. So you can use them as if they're "normal" functions.

Illfounded

The question was "how does that work?", not "I need a cross-platform solution for insert-problem-name-here". – Markley

The OP want to understand what dynamic linking is. He has not asked us "who do I Abstract LoadLibrary()". – Churlish

@jalf it's answer to the quetion; it abstracts platforms' shared library. – Illfounded

Oh my!! Sorry for my foolishness – Illfounded

I still think that it isn't explained really well... "on windows, there is .lib file to use .dll. When you link to this .lib file, you don't need to call LoadLibrary and GetProcAddress". Don't other OS provide the same functionality? "And if you write code such as Foo();, compiler convert it into (*Foo)();". Does it? Doesn't the loader do that? And the function pointer analogy isn't perfect either, because the call ends up being as static as any other calls. By saying that I mean that statically linked DLLs are slightly faster to call at runtime than that dlopened. – Markley

@Markley The choice between whether you need to call LoadLibrary or not is actually made before you link. If you use a symbol from the library in your code, you do not have to call LoadLibrary. And of course, Unix provides the same functionality, with the same distinction: if you use the symbol, you don't need to call dlopen, but you do need to link against the .so. (It was, of course, totally unnecessary for Windows to require two different files, since all of the information in the .lib is present in the .dll.) – Saltatorial

@Markley And the compiler certainly doesn't translate Foo() into (*Foo)(). Nor does the loader, for that matter; as you point out, there is no pointer. Very roughly speaking: the compiler will insert a placemarker for Foo in the object file, and the linker will fill it in with the actual address. The difference between static linking and dynamic is when this filling in occurs. – Saltatorial