Which is the correct way to build a static library with link time code generation on Linux?

Question

I'm wandering which is the right way to compile static library on Linux with GCC in such way that when link time optimizations (LTO) be applied to the executable the library to be consumable and possibly to be achieved optimal performance.

When library is compiled only with -flto the executable cannot be linked to it no matter whether it uses -flto or not. The error is:

undefined reference to `hello'

where hello is function defined in the library.

According to the answer to this Stack Overflow question one possible solution is the following:

set(CMAKE_AR gcc-ar)
set(CMAKE_C_ARCHIVE_CREATE "<CMAKE_AR> qcs <TARGET> <LINK_FLAGS> <OBJECTS>")
set(CMAKE_C_ARCHIVE_FINISH true)

Then the library can be linked to the executable both with -flto and without -flto passed to the linker flags.

But according to the answer to this Stack Overflow question if we want static library to be compiled in such way to be usable with and without LTO than we have to use -ffat-lto-objects. If we add this flag to the library compilation flags again the library can be linked to executables both with -flto and without -flto passed to the linker flags.

My questions are:

1. What is the exact meaning of the first solution with gcc-ar?

2. What is the difference between different working variants when library is compiled with -flto.

   2.1 Executable without -flto.

   • Library is using only gcc-ar.
   • Library is using only -ffat-lto-objects.
   • Library is using both gcc-ar and -ffat-lto-objects

   2.2 Executable with -flto and the same 3 variants for the library.

Here is Minimal, Complete, and Verifiable example with my test project which is modified version of the example from this Stack Overflow question. I'm using GCC version 7.2.0.

CMakeLists.txt

cmake_minimum_required(VERSION 3.9)

project(lto)

set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS_RELEASE} -g -O3")

add_subdirectory(lib)
add_subdirectory(exe)

exe/CMakeLists.txt

set(TARGET_NAME exe)

add_executable(${TARGET_NAME} src/main.c)

target_link_libraries(${TARGET_NAME} lib)

option(EXE_LTO "Use link time optimizations for the executable." OFF)

if(${EXE_LTO})
  target_compile_options(${TARGET_NAME} PRIVATE "-flto")
endif()

exe/src/main.c

extern void hello();

int main()
{
  hello();
  return 0;
}

lib/CMakeLists.txt

set(TARGET_NAME lib)

add_library(${TARGET_NAME} STATIC src/lib.c)

option(LIB_LTO "Use link time optimizations for the library." OFF)
option(LIB_FAT_LTO "Create fat LTO objects for library files." OFF)
option(LIB_USE_LTO_AR "Use another AR program for LTO objects." OFF)

if(${LIB_LTO})
  target_compile_options(${TARGET_NAME} PRIVATE -flto)
endif()

if(${LIB_FAT_LTO})
  target_compile_options(${TARGET_NAME} PRIVATE -ffat-lto-objects)
endif()

if(${LIB_USE_LTO_AR})
  set(CMAKE_AR gcc-ar)
  set(CMAKE_C_ARCHIVE_CREATE "<CMAKE_AR> qcs <TARGET> <LINK_FLAGS> <OBJECTS>")
  set(CMAKE_C_ARCHIVE_FINISH true)
endif()

lib/src/lib.c

#include <stdio.h>

void hello()
{
  puts("Hello");
}

Concoff

I suggest you to add cmake tag to your question. – Jidda

@LethalProgrammer CMake is not really relevant to the question but I only use it as build system. All of this can be done also without CMake. – Concoff

"the executable cannot be linked to it". What error messages are you getting? – Volleyball

@n.m. Undefined references to called lib functions. – Concoff

Seems unlikely. Can you produce a minimal reproducible example? – Volleyball

@n.m. I added it to the question. – Concoff

Answer 1

If you don't add --plugin /psth/to/lto-plugin.so* to ar parameters, you will get undefined references at link time and a warning at archive creation time. Or at least that's what I'm getting. You probably need to add it here:

set(CMAKE_C_ARCHIVE_CREATE "<CMAKE_AR> --plugin <LTO_PLUGIN_PATH> \
  qcs <TARGET> <OBJECTS>")

(LINK_FLAGS probably don't belong here, so I omitted them).

I don't know how to set LTO_PLUGIN_PATH automatically.

The plugin enables ar to create LTO-enabled archives. All other methods either don't work at all (non-fat objects in the archive) or don't actually result in link time optimisation (fat objects in the archive — in this case only the traditional object code is used, the LTO information is ignored).

Volleyball

This doesn't answer the questions which I asked. What is the meaning of liblto_plugin used by ar and what is the difference of this solution with using only -ffat-lto-objects which also works without liblto_plugin? See My questions are: part. – Concoff

@Concoff the meaning of --pkugun flag is "use the plugin". It is required for building lto-enabled archives, as your warning message should have said. – Volleyball

But why the example works also only with -ffat-lto-objects? Maybe than always normal compiled code is used by linker instead of intermediate code no matter of -flto flag? – Concoff

"All other methods either don't work at all (non-fat objects in the archive) or don't actually result in link time optimisation (fat objects in tge archive)." I think that this answers the question and I will accept it. – Concoff

I think you can place a symlink to the plugin .so in /usr/lib/bfd-plugins/ and ar, nm etc should pick it up (but I'm struggling with LTO at the moment myself) – Pokeberry

Answer 2

Instructions:

1. Compile your object files with options -flto -fno-fat-lto-objects. -fno-fat-lto-objects is not strictly necessary, however, it makes sure it either does lto or fails (instead of falling back to non-lto mode).
2. Create the static library with ar rcsT --plugin <path-to-lto-plugin> ... You need the full path to liblto_plugin.so here. Option T here creates a thin archive (does not copy .o files).
3. Link with -flto flag.

Example:

$ cat library.cc
namespace library {

int f(int a) {
    return a + 1;
}

}
$ cat test.cc
#include <iostream>

namespace library { int f(int); }

int main() {
    std::cout << library::f(0) << '\n';
}
$ g++ -c -Wall -Wextra -std=gnu++14 -O3 -flto -fno-fat-lto-objects -o library.o library.cc
$ ar rcsT --plugin /usr/libexec/gcc/x86_64-redhat-linux/5.3.1/liblto_plugin.so library.a library.o 
$ g++ -c -Wall -Wextra -std=gnu++14 -O3 -flto -fno-fat-lto-objects -o test.o test.cc
$ g++ -o test -flto -O3 test.o 
/tmp/ccY0l2jQ.ltrans0.ltrans.o: In function `main':
<artificial>:(.text.startup+0x37): undefined reference to `library::f(int)'
collect2: error: ld returned 1 exit status
$ g++ -o test -flto test.o library.a
$ ./test
1

Deforest

This doesn't answer the questions which I asked. What is the meaning of liblto_plugin used by ar and what is the difference of this solution with using only -ffat-lto-objects which also works without liblto_plugin. See My questions are: part. – Concoff