Linking Windows DLL files from static libraries using CMake without hand-crafting unresolved symbol names

Question

The Situation

I'm using Visual Studio 2008 SP1 (Professional Edition, both for 32-bit and 64-bit builds). I'm seeking a workaround to what I believe is a very unhelpful "limitation" in Visual Studio.

I find it quite surprising that the Visual Studio linker and compiler does not do this right at DLL file creation time, to automatically scan all specified static libraries for all exported symbols in the same manner given in Building an Import Library and Export File and in a StackOverflow comment. I confirmed that it is not sufficient to simply apply __declspec(dllexport) and __declspec(dllimport) attributes to class, function, and data declarations in the files that comprise the static libraries.

The linker does not scan all of the static libraries for exported symbols and so won't pull them into the DLL file (the symbols have to be referenced by .obj files on the DLL link command-line or by other means I show below). Without explicit references to each exported symbol, the DLL file may still be created, but its associated import library file is not created.

From what I can gather, Microsoft is recommending using LIB.EXE to create the DEF file, but unfortunately, the LIB.EXE page applies a constraint:

Note that if you create your import library in a preliminary step, before creating your .dll, you must pass the same set of object files when building the .dll, as you passed when building the import library.

That is an unfortunate constraint given that I am also using CMake in my new build environment. CMake hides the details of what is actually passed to the linker (and I deem that to be a good thing in 99% of the time), but in this case I need to get access to some of that information at CMake execution time, not afterwards using hand-crafted scripts or other fragile skulduggery.

The Questions:

How do I go about forcing the DLL linker to resolve all exported symbols in all static libraries that comprise the DLL file, that isn't going to result in fragility and additional build logic maintenance chores? Think in terms of full-automation here, and keep in mind that I need to do this multiple times for multiple different DLL's.

My questions are:

1. How do I obtain the set of object files and static libraries used on the final DLL link command line using CMake syntax alone?

2. Does the order of files listed on LIB.EXE line (the one used to generate the DEF file) have to match exactly that order used on the DLL link line?

3. Are there other difficulties that I might encounter from the use of the LIB.EXE approach to generate the DEF file?

4. Is there a better way to go about this that does not require calling a separate utility, such LIB.EXE, before calling the final link? I'm concerned about the added build overhead beyond the link itself for LIB.EXE to rescan all of the static libraries again even though it just wrote them out in separate executions.

The Non-Answers:

The following are solutions that I cannot consider right now:

1. Manually specifying the unreferenced symbols anywhere other than in the original .h or .cpp files, as doing that is going to break each time a developer forgets to update the file that lists the (quite possibly name-mangled) symbol name. And it will break with non-user-friendly linker errors about unresolved symbols which will be costly for developers to debug. This non-answer includes the following approaches:

   1. Explicitly added .obj files to the DLL link command-line, (variants of this include adding "fake" .obj files that have dummy references to the otherwise unreferenced but exported symbols (and note that this is what my old build environment does today, and it stinks)), and,

   2. Handcrafting DEF files to include the unreferenced but exported symbols, and,

   3. Linker command-line options that specifically reference the unreferenced but exported symbols symbols.

2. Stop using static libraries altogether. I cannot do that in the short-term, as it is too much of a structural change from the old build environment I am moving away from. It is quite likely I will take this route in the future, once all remnants of the old build environment are in the "trash bin", but that is not my focus here.

Reference material:

• Exporting from a DLL Using DEF Files
• Module Definition (.def) Files
• PRB: Can't Export from Static Libraries w/_declspec(dllexport)
• dumpbin.exe /symbols option
• lib.exe options
• Building an Import Library and Export File
• CMake LINK_FLAGS target property

Wrangle

The Mozilla build system uses non-answer 1.1 and every so often someone has to remember to update the dlldeps.cpp file so that the right things get exported. Their plan is to switch to non-answer 2. – Woolworth

I can't help, but that's one of the most well-written questions I've ever seen. Best of luck. – Steffy

Part of the dll linker's job is to not include stuff that doesn't need to be included: i.e. unreferenced functions in static libraries. So, unless you explicitly reference a function in a static lib from a function that is dllexport'ed, it should not even be included in the final dll, let alone exported. I think the real answer for you is non-answer 2, however painful it may be to get there... I think it will prove less painful than maintaining any sort of automatic synchronization framework you might come up with as a custom build tool solution. – Zumstein

Thanks everyone. I'm going to live with non-answer #1.1 for now until I have the budget to implement non-answer #2, as #2 just feels like the right way to go. – Wrangle

Answer 1

I'm only answering to make it clear your feeling to not use static libraries is correct.

DLRdave said it already in the comments, your build system is being abused by LIB files. A LIB file is much like a real library, you only walk out with the things you ask for, not everything in the library.

If there's a gap in the Visual Studio 2008 tool set, it's that it doesn't support partial linking. The input into a partial link is a set of OBJ files and the output is a single OBJ file that contains all the code and data from the input OBJ files.

The difference between an archive/library and a partial link is described for g++ in the answer to this question: g++ partial linking instead of archives?, where the GNU linker (ld) does support partial linking.

As for possible short term mitigation - personally, I would have tried to use scripting to dynamically build the DEF file at build time by using LIB /List or DUMPBIN /ARCHIVEMEMBERS to get the obj files and LIB /DEF to generate the DEF file from that list. Or, as I assume _declspec(dllexport) is being used, you could have also used DUMPBIN /DIRECTIVES, looked for /EXPORT and built the DEF file yourself.

Probationer

I agree with most of that. Now that I have worked with CMake a bit more, it might be possible to create a special wrapper around calls to add_library(... STATIC ...) to implement such a DEF generation mechanism. I'll have to give that some more thought. – Wrangle