How to write LTO-enabled code?

Question

What are the caveats and gotchas needed to keep in mind when writing code or build scripts for compilation with LTO?

The motivation behind this question is to understand better why some projects do not compile cleanly when LTO is enabled. In particular, I could not build ICU with LTO enabled, neither in MSVC nor in GCC. In other cases, I can enable LTO with a given toolchain version but not with another (newer) version; this happens, for example, with libiconv.

In all failed cases I met, a linking failure due to unresolved symbols is involved.

Why exactly does this happen? Is this a problem with the toolchain, with the build script or with the source code?

Sheridansherie

I've never encountered a project that didn't compile with LTCG enabled in MSVC. What error messages are you getting when you try to build ICU in MSVC? The same settings you use to compile an optimized build should also work when you flip the LTCG switch. In fact, LTCG is turned on by default for optimized builds in the built-in project template. – Shah

Encapsulation. Reduce the quantity of global variables. Reduce external dependencies in functions. Use const as appropriate. – Thrombin

@CodyGray: I get undefined symbols at link time (_uconvmsg_dat in some cases, _icudt58_dat in others). I'm using several build flags for that matter (/MT /Oxy- /fp:fast /Zi), however the build always passes without LTCG (/GL and /LTCG). – Sheridansherie

@CodyGray: I've added an answer that explains why ICU doesn't build correctly in MSVC (32-bit) with LTCG enabled, it might be of your interest to know why. In this case it is indeed a defect of the package and not of the toolchain. – Sheridansherie

Answer 1

This answer summarizes my findings with regards to some of the complications involved when building projects with LTO enabled, both in GCC and in MSVC.

GCC

First of all, as per the GCC Wiki, in order to properly build a project with LTO enabled, you must:

1. Make sure gcc-ar is used instead of binutils ar;
2. Make sure gcc-ranlib is used instead of binutils ranlib;
3. Make sure gcc-nm is used instead of binutils nm;
4. Compile and link with -flto.

This means that in the traditional ./configure && make cycle, one must mind setting the value of AR=, RANLIB= and NM= when relevant. That's mostly it; however, these steps are easily overlooked, because needing to change the value of eg. AR is rather uncommon.

Now to the issues:

In GCC 4.8 and earlier versions, the compiler emits fat object files as default. This means that even if the post-compilation tools (linker, archiver, etc.) do not recognize LTO objects, they'll work normally (but without actually performing LTO).

In GCC 4.9 and later versions, the compiler emits slim object files as default, meaning that post-compilation tools must recognize LTO objects, or the tools will fail. This explains why sometimes an LTO build passes when using GCC 4.8, but fails when using GCC 4.9 and later.

I have also noticed that build scripts do not always pass correctly the values of some configuration directives to sub-scripts when needed. For example, when building static libiconv with LTO in MinGW-w64, the configure script still configures the internal libtool with ar instead of gcc-ar, even when told that AR=gcc-ar.

LTO builds tend to uncover hidden errors, in particular errors caused by static init order fiasco. They can also get in the way of other optimizations, such as ICF (performed by Gold).

Lastly, there are still a number of bugs apparently persisting in the LTO machinery. When attempting to compile ICU with MinGW-w64 with LTO and other optimizations enabled, I've faced this bug and internal compiler errors (internal compiler error: in splice_child_die, at dwarf2out.c, possibly related to using -g with LTO).

All of this means that due to several imperfections in the toolchain, building a random project with LTO can still be non-trivial. Some projects will build successfully and some others won't.

MSVC

To compile with LTO in MSVC (which is called LTCG), /GL must be used when compiling and /LTCG must be used when linking, and that's really it.

Nonetheless, when LTCG is enabled in MSVC, the compiler does not emit traditional COFF objects. It emits instead a special kind of object file containing IR whose header (ANON_OBJECT_HEADER_BIGOBJ) is different from the COFF header (IMAGE_FILE_HEADER). This obviously should make no difference at all when building a project, since these details are left for the toolchain to handle.

Now, why doesn't ICU build properly when LTCG is enabled in MSVC?

ICU has a tool called pkgdata which generates object code for a given architecture. During the build process, the tool is used to build other utilities in the package. However, pkgdata tries to guess the target architecture by inspecting a given reference object file. In Windows, the tool assumes a COFF header, and in 32-bit builds it wrongly determines that a 64-bit architecture is being targeted (due to the sloppy logic inside pkg_genc.c:getArchitecture()). Hence the MSVC 32-bit LTCG build fails.

Sheridansherie

Thank you for your Answer and sharing your experience. What about LTO and Clang and LLVM toolchain? – Armstrong

@GeoObserver: I had fewer problems with the LLVM machinery in this aspect (although I've been working with LLVM exclusively on Linux). Compiling and linking with -flto has been enough so far. – Sheridansherie

the MSVC docs are pretty useful too learn.microsoft.com/en-us/cpp/build/reference/… – Ichinomiya