How do I merge multiple PDB files?

Question

We are currently using a single command line tool to build our product on both Windows and Linux.

Si far its works nicely, allowing us to build out of source and with finer dependencies than what any of our previous build system allowed. This buys us great incremental and parallel build capabilities.

To describe shortly the build process, we get the usual:

.cpp -- cl.exe --> .obj and .pdb
multiple .obj and .pdb -- cl.exe --> single .dll .lib .pdb
multiple .obj and .pdb -- cl.exe --> single .exe .pdb

The msvc C/C++ compiler supports it adequately.

Recently the need to build a few static libraries emerged. From what we gathered, the process to build a static library is:

multiple .cpp -- cl.exe --> multiple .obj and a single .pdb
multiple .obj -- lib.exe --> a single .lib

The single .pdb means that cl.exe should only be executed once for all the .cpp sources. This single execution means that we can't parallelize the build for this static library. This is really unfortunate.

We investigated a bit further and according to the documentation (and the available command line options):

• cl.exe does not know how to build static libraries
• lib.exe does not know how to build .pdb files

Does anybody know a way to merge multiple PDB files ? Are we doomed to have slow builds for static libraries ? How do tools like Incredibuild work around this issue ?

Document

You could always put your code on a solid state drive. Builds will be lightning fast. – Jahvist

I'm on a solid state drive on my stylebook. It helps, but linking is IO bound, compiling is CPU bound. – Stockyard

Answer 1

I've not done C++ for a long time but from this article, it appears that this is a performance trick to stop the recreation of symbols for common headers.

You could try /Z7 to embed info in each obj, and not create a PDB and then link and recreate it with a rebase as in this article.

Stanza

Isn't the debug information from /Z7 different (and inferior) from what is generated by /Zi and /ZI? According to the article linked to in the egghead cafe thead (support.microsoft.com/kb/258205) you can extract the debug info from /Z7 into a .dbg file and not a .pdb file. – Stockyard

both of these links are now dead :( – Fonsie

Answer 2

No need to merge PDB files.

Compile source files with /Z7 to avoid creating a PDB during the CL.EXE steps.

Use LIB.EXE to create static libaries with embedded debugging information. Use LINK.EXE instead of CL.EXE to link, use /PDB to specify where the debugging information goes.

If you are debugging a process with an EXE and one or more DLLs, feed your debugger a PDB for each image (EXE or DLL).

Padrone

Answer 3

Merge PDB files are possible but only can be done by cl.exe and link.exe. I do NOT know any standlone tools to merge PDB files.

You can use /PDB option to linker(I checked VC2005) to specify alternate pdb file name.

Microsoft suggests to also include PDB files(each obj has a corresponding PDB file) along with .LIB file.

You cannot archive PDB files inside .LIB file, I've tried it with VC2003, failed.

Compile with /Z7 can avoid PDB files for .LIB, but object files are large, unless the link.exe strip the debug information. If you have no /debug option to linker, then your exe/dll cannot be debugged.

Compiler(cl.exe) always write to vcXX.pdb file unless you use /Fd option to specify another name. Even when you use cl.exe to produce an executable "directly", It'll produce a vc80.pdb file and then the link.exe will produce the pdb file name same as the executable.

cl /Zi test.c

cl.exe -> vc80.pdb link.exe read vc80.pdb(the name is embedded in test.obj file) -> test.pdb

Each time cl /Zi /c compile a file, it'll try to modify the existing vcXX.pdb file instead of overwrite it.

I got the above conslusion by play with the compiler again and again, then capture sysinternals's procexp result and analyze it. Hope it helps.

Heartbroken

Answer 4

Unless you want to redistribute the static libraries with debug info, you don't actually need to merge any PDB files (or use /Z7 to embed the debug info).

As @zhaorufei mentioned, when using /Zi, each object file contains a reference to its PDB file, which the linker then uses.

Simply use /Fd to give each object a unique PDB file:

> cl -c foo.cpp -Fo:target/foo.obj -Fd:target/foo.pdb -Zi
> cl -c bar.cpp -Fo:target/bar.obj -Fd:target/bar.pdb -Zi

> strings target/foo.obj | grep pdb
D:\Dev\sample\target\foo.pdb
> strings target/bar.obj | grep pdb
D:\Dev\sample\target\bar.pdb

This also has the benefit that it works around the issues of concurrent access to shared PDB files mentioned here, so you can parallelize the compile step like you wanted.

Then link/archive the object files as usual. VC++ already embeds various kinds of information in the object files to pass them to the linker, such as the runtime link setting and dependency libraries - the PDB file path is no different. Creating a static library from the objects does not remove the references:

> lib -out:target/all.lib target/foo.obj target/bar.obj
> strings target/all.lib | grep pdb
D:\Dev\sample\target\bar.pdb
D:\Dev\sample\target\foo.pdb

When linking this library to an executable or DLL, the linker still pulls in the debug info from the referenced PDBs and adds it to the final PDB file.

The only caveat I can see is that the path is always absolute, so this may not work if you move the files around locally or to another machine before linking.

Holdall