Is using --start-group and --end-group when linking faster than creating a static library?

Question

If one builds static libraries in one's build scripts and one wants to use those static libraries in linking the final executable, the order one mentions the .a files is important:

g++ main.o hw.a gui.a -o executable

If gui.a uses something defined in hw.a the link will fail, because at the time hw.a is processed, the linker doesn't yet know that the definition is needed later, and doesn't include it in the being.generated executable. Manually fiddling around with the linker line is not practical, so a solution is to use --start-group and --end-group which makes the linker run twice through the libraries until no undefined symbols are found anymore.

g++ main.o -Wl,--start-group hw.a gui.a -Wl,--end-group -o executable

However the GNU ld manual says

Using this option has a significant performance cost. It is best to use it only when there are unavoidable circular references between two or more archives.

So I thought that it may be better to take all .a files and put them together into one .a file with an index (-s option of GNU ar) which says in what order the files need to be linked. Then one gives only that one .a file to g++.

But I wonder whether that's faster or slower than using the group commands. And are there any problems with that approach? I also wonder, is there better way to solve these interdependency problems?

---

EDIT: I've written a program that takes a list of .a files and generates a merged .a file. Works with the GNU common ar format. Packing together all static libs of LLVM works like this

$ ./arcat -o combined.a ~/usr/llvm/lib/libLLVM*.a

I compared the speed against unpacking all .a files manually and then putting them into a new .a file using ar, recomputing the index. Using my arcat tool, I get consistent runtimes around 500ms. Using the manual way, time varies greatly, and takes around 2s. So I think it's worth it.

Code is here. I put it into the public domain :)

Albion

Answer 1

You can determine the order using the lorder and tsort utilities, for example

libs='/usr/lib/libncurses.a /usr/lib/libedit.a'
libs_ordered=$(lorder $libs | tsort)

resulting in /usr/lib/libedit.a /usr/lib/libncurses.a because libedit depends on libncurses.

This is probably only a benefit above --start-group if you do not run lorder and tsort again for each link command. Also, it does not allow mutual/cyclic dependencies like --start-group does.

Leyva

I was trying to find lorder for linux prior to asking this question, but I couldn't find it. The only few manpages I found on the internet said that lorder is deprecated and that ar replaces it. I couldn't find WTF command I have to give to ar so it does the same as lorder. – Albion

Debian and Ubuntu have it in the package bsdmainutils. – Leyva

Answer 2

Is there a third option where you just build a single library to begin with? I had a similar problem and I eventually decided to go with the third option.

In my experience, group is slower than just unifying the .a files. You can extract all files from the archive, then create a new .a file from from the smaller files

However, you have to be careful about a circumstance where both files happen to contain the same definition (you can explicitly check for this by using nm to see what definitions are contained in each library)

Jibe

+1 there is absolutely no sense in making two libraries when the libraries have a mutual dependency on one another. – Afeard

@R.., I don't understand. The world is full of libraries that depend on other libraries. Take clang as an example. Its libsema depends on its libast. Melting them into a single library wouldn't be good. The whole point of having libraries separate is to not drag a completely irrelevant functionality into another library. – Albion

Even if libB depends on libA, it's fine for them to be separate libraries, as long as libA can be used independently of libB. But if there's a mutual dependency between the two, it's purely a nuisance to separate them. – Afeard

@R.. there is not necessarily a mutual dependency in the case I describe. I don't want to modify build scripts/makefiles to change the order of .a files when modifying source files to add a dependency. In a multi-person project, the people programming aren't even necessarily involved with managing the build files. – Albion

@Johannes To do what you want, you have to extract all of the o files (using ar x) and then recombine (using another ar command). But you should make sure that you dont accidentally duplicate any entries (for example, if you have an inline function, and the compiler decides not to inline it for multiple source files, you will have a definition of that inline function in multiple .o files) – Jibe

@Foo I wonder whether there is a tool which can concatenate multiple .a files into one? The way I see it, it would be much more performant to reuse the symbol tables already present in the individual .a files, and just concatenating the symbol list / file name list contained into the .a files instead of recomputing them for the final .a file. – Albion

The reason why you can't just merge the files is because you would have to separately merge the indices and merge the object files. In particular, the former is very difficult: the index contains information like external dependencies which aren't simply resolved by catting. You could do it if your build process didn't use ar r (like in older build patterns, which run ranlib at the end. – Jibe

@FooBah my arcat seems to work. You don't need to resolve anything. You just need to state in the index where symbols are found (in what archive member). – Albion

Mutual dependency does not necessarily mean the libraries may always be merged. One (might be the only) use case I have encountered is if the libraries are going to provide symbols that are not unique in order to create link-time project configuration (sort of module-based behaviour, where the functionality DoFoo() might be provided by linking against libFooBar.a or libFooBaz.a). – Acaudal