CMake: How to build external projects and include their targets

Question

I have a Project A that exports a static library as a target:

install(TARGETS alib DESTINATION lib EXPORT project_a-targets)
install(EXPORT project_a-targets DESTINATION lib/alib)

Now I want to use Project A as an external project from Project B and include its built targets:

ExternalProject_Add(project_a
  URL ...project_a.tar.gz
  PREFIX ${CMAKE_CURRENT_BINARY_DIR}/project_a
  CMAKE_ARGS -DCMAKE_INSTALL_PREFIX:PATH=<INSTALL_DIR>
)

include(${CMAKE_CURRENT_BINARY_DIR}/lib/project_a/project_a-targets.cmake)

The problem is that the include file does not exist yet when CMakeLists of Project B is run.

Is there a way to make the include dependent on the external project being built?

Update: I wrote a short CMake by Example tutorial based on this and other common problems I encountered.

Frowsy

Answer 1

I think you're mixing up two different paradigms here.

As you noted, the highly flexible ExternalProject module runs its commands at build time, so you can't make direct use of Project A's import file since it's only created once Project A has been installed.

If you want to include Project A's import file, you'll have to install Project A manually before invoking Project B's CMakeLists.txt - just like any other third-party dependency added this way or via find_file / find_library / find_package.

If you want to make use of ExternalProject_Add, you'll need to add something like the following to your CMakeLists.txt:

ExternalProject_Add(project_a
  URL ...project_a.tar.gz
  PREFIX ${CMAKE_CURRENT_BINARY_DIR}/project_a
  CMAKE_ARGS -DCMAKE_INSTALL_PREFIX:PATH=<INSTALL_DIR>
)

include(${CMAKE_CURRENT_BINARY_DIR}/lib/project_a/project_a-targets.cmake)

ExternalProject_Get_Property(project_a install_dir)
include_directories(${install_dir}/include)

add_dependencies(project_b_exe project_a)
target_link_libraries(project_b_exe ${install_dir}/lib/alib.lib)

Bullfinch

Thanks for your answer. What you suggest is similar to what I had before. I hoped to find a way to make use of the exported targets as it seemed like a nicer interface than specifying the lib paths manually... – Frowsy

@mirkok In some ways I agree that it's a nicer interface. There are a couple of other options. You could for example just include the source of Project A in a subdirectory of Project B and pull it in via add_subdirectory. Or you could use ExternalProject_Add and do some trickery which causes CMake to run twice; the first time building the external project, the second time successfully picking up its import file "project_a-targets.cmake". – Bullfinch

I wanted to avoid having to include the source of external projects in my source tree. It would be great if ExternalProject_Add just behaved like add_subdirectory and exposed all targets. The solution you described above is probably still the cleanest. – Frowsy

Consider making them both ExternalProject builds, and then have B depend on A, and then the CMakeLists file for project B would include the targets file from project A, but your "Super Build" CMakeLists would just build A and then B, both as ExternalProjects... – Butyraceous

@Butyraceous - I've seen the Super Build solution recommended a few times, but I guess I'm not sure what benefits it provides over only including some external projects via ExternalProject. Is it consistency, or more canonical, or something else? I'm sure I'm missing something fundamental here. – Bullfinch

One of the problems with this solution is that we've just hardcoded the library name (alib.lib), which makes the build system not cross-platform, as different OS use different naming schemes for shared libraries, and adapting to these different naming schemes is one of the features of CMake. – Veer

@Nate - that's true, but there's not always a particularly elegant way round it that isn't discussed here. As DLRdave commented, we could make both projects ExternalProject builds then we could use the installed CMake file of ProjectA to avoid this ugliness. It's not just the cross-platform side of things which is an issue either - it can be a struggle to handle different build types too when resulting libraries have the same name in Debug and Release. See for example the bottom of the code in this answer. – Bullfinch

I'm still a bit confused on what you mean by "two different paradigms", it sounds like you're saying that include happens at Configure time (and executes the include script immediately) but ExternalProject_Add merely adds an external CMakeScript to the list of available projects, and only executes it the script at uh... Build time? Or is it true to say that ExternalProject_Add only runs the CMakeLists script when it's needed as a dependency? Am I on the right track here? – Athletics

I read some more articles and thought about it a bit more. E.g., if you had ExternalProject_Add, that pulled a repository containing a CMakeLists.txt from github, ExternalProject_Add only git clones and runs the CMakeLists.txt when the main project is built. When the main project is configured or generated, CMake does not download, configure or generate ExternalProjects. I found it a bit confusing, but for an ExternalProject, configure, generate, and build all happen when the main project is built. – Athletics

I Don't understand how is this an accepted answer - ${install_dir}/lib/alib.lib missing and no known rule to make it @Bullfinch – Monasticism

Answer 2

This post has a reasonable answer:

CMakeLists.txt.in:

cmake_minimum_required(VERSION 2.8.2)

project(googletest-download NONE)

include(ExternalProject)
ExternalProject_Add(googletest
  GIT_REPOSITORY    https://github.com/google/googletest.git
  GIT_TAG           master
  SOURCE_DIR        "${CMAKE_BINARY_DIR}/googletest-src"
  BINARY_DIR        "${CMAKE_BINARY_DIR}/googletest-build"
  CONFIGURE_COMMAND ""
  BUILD_COMMAND     ""
  INSTALL_COMMAND   ""
  TEST_COMMAND      ""
)

CMakeLists.txt:

# Download and unpack googletest at configure time
configure_file(CMakeLists.txt.in
               googletest-download/CMakeLists.txt)
execute_process(COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}" .
  WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/googletest-download )
execute_process(COMMAND ${CMAKE_COMMAND} --build .
  WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/googletest-download )

# Prevent GoogleTest from overriding our compiler/linker options
# when building with Visual Studio
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)

# Add googletest directly to our build. This adds
# the following targets: gtest, gtest_main, gmock
# and gmock_main
add_subdirectory(${CMAKE_BINARY_DIR}/googletest-src
                 ${CMAKE_BINARY_DIR}/googletest-build)

# The gtest/gmock targets carry header search path
# dependencies automatically when using CMake 2.8.11 or
# later. Otherwise we have to add them here ourselves.
if (CMAKE_VERSION VERSION_LESS 2.8.11)
  include_directories("${gtest_SOURCE_DIR}/include"
                      "${gmock_SOURCE_DIR}/include")
endif()

# Now simply link your own targets against gtest, gmock,
# etc. as appropriate

However it does seem quite hacky. I'd like to propose an alternative solution - use Git submodules.

cd MyProject/dependencies/gtest
git submodule add https://github.com/google/googletest.git
cd googletest
git checkout release-1.8.0
cd ../../..
git add *
git commit -m "Add googletest"

Then in MyProject/dependencies/gtest/CMakeList.txt you can do something like:

cmake_minimum_required(VERSION 3.3)

if(TARGET gtest) # To avoid diamond dependencies; may not be necessary depending on you project.
    return()
endif()

add_subdirectory("googletest")

I haven't tried this extensively yet but it seems cleaner.

Edit: There is a downside to this approach: The subdirectory might run install() commands that you don't want. This post has an approach to disable them but it was buggy and didn't work for me.

Edit 2: If you use add_subdirectory("googletest" EXCLUDE_FROM_ALL) it seems means the install() commands in the subdirectory aren't used by default.

Djambi

This is probably just me being overly cautious because this is just an example and gtest is probably pretty stable, but I strongly recommend always using a specific GIT_TAG during clone, you could lose build repeatability because 2 years from now somebody running the build script will get a different version than what you did. CMake's docs recommend this too. – Athletics

Why include(ExternalProject) is it not part of CMake keywords by default? – Brevier

Answer 3

Edit: CMake now has builtin support for this. See new answer which uses FetchContent.

You can also force the build of the dependent target in a secondary make process

See my answer on a related topic.

Shipp

Answer 4

I was searching for similar solution. The replies here and the Tutorial on top is informative. I studied posts/blogs referred here to build mine successful. I am posting complete CMakeLists.txt worked for me. I guess, this would be helpful as a basic template for beginners.

"CMakeLists.txt"

cmake_minimum_required(VERSION 3.10.2)

# Target Project
project (ClientProgram)

# Begin: Including Sources and Headers
include_directories(include)
file (GLOB SOURCES "src/*.c")
# End: Including Sources and Headers


# Begin: Generate executables
add_executable (ClientProgram ${SOURCES})
# End: Generate executables


# This Project Depends on External Project(s) 
include (ExternalProject)

# Begin: External Third Party Library
set (libTLS ThirdPartyTlsLibrary)
ExternalProject_Add (${libTLS}
PREFIX          ${CMAKE_CURRENT_BINARY_DIR}/${libTLS}
# Begin: Download Archive from Web Server
URL             http://myproject.com/MyLibrary.tgz
URL_HASH        SHA1=<expected_sha1sum_of_above_tgz_file>
DOWNLOAD_NO_PROGRESS ON
# End: Download Archive from Web Server

# Begin: Download Source from GIT Repository
#    GIT_REPOSITORY  https://github.com/<project>.git
#    GIT_TAG         <Refer github.com releases -> Tags>
#    GIT_SHALLOW     ON
# End: Download Source from GIT Repository

# Begin: CMAKE Comamnd Argiments
CMAKE_ARGS      -DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_CURRENT_BINARY_DIR}/${libTLS}
CMAKE_ARGS      -DUSE_SHARED_LIBRARY:BOOL=ON
# End: CMAKE Comamnd Argiments    
)

# The above ExternalProject_Add(...) construct wil take care of \
# 1. Downloading sources
# 2. Building Object files
# 3. Install under DCMAKE_INSTALL_PREFIX Directory

# Acquire Installation Directory of 
ExternalProject_Get_Property (${libTLS} install_dir)

# Begin: Importing Headers & Library of Third Party built using ExternalProject_Add(...)
# Include PATH that has headers required by Target Project
include_directories (${install_dir}/include)

# Import librarues from External Project required by Target Project
add_library (lmytls SHARED IMPORTED)
set_target_properties (lmytls PROPERTIES IMPORTED_LOCATION ${install_dir}/lib/libmytls.so)
add_library (lmyxdot509 SHARED IMPORTED)
set_target_properties(lmyxdot509 PROPERTIES IMPORTED_LOCATION ${install_dir}/lib/libmyxdot509.so)

# End: Importing Headers & Library of Third Party built using ExternalProject_Add(...)
# End: External Third Party Library

# Begin: Target Project depends on Third Party Component
add_dependencies(ClientProgram ${libTLS})
# End: Target Project depends on Third Party Component

# Refer libraries added above used by Target Project
target_link_libraries (ClientProgram lmytls lmyxdot509)

Latticed

Answer 5

I would suggest to use FetchContent first:

include(FetchContent)
FetchContent_Declare(glog
    GIT_REPOSITORY    https://github.com/google/glog.git
    SOURCE_DIR        ${CMAKE_CURRENT_SOURCE_DIR}/third_party/glog
    SUBBUILD_DIR      third_party/glog/subbuild
    BINARY_DIR        third_party/glog/build
)
option(WITH_GFLAGS "" OFF)
option(WITH_GTEST "" OFF)
option(WITH_GMOCK "" OFF)
option(WITH_UNWIND "" OFF)
option(BUILD_SHARED_LIBS "" OFF)
option(BUILD_TESTING "" OFF)
FetchContent_MakeAvailable(glog)

....

add_library(libsomething STATIC)
target_link_libraries(libsomething PUBLIC glog::glog)

Note, all CMake magic works as expected: you need not specify glog's include directory or build artifacts for libsomething.

Mesentery

Answer 6

cmake's ExternalProject_Add indeed can used, but what I did not like about it - is that it performs something during build, continuous poll, etc... I would prefer to build project during build phase, nothing else. I have tried to override ExternalProject_Add in several attempts, unfortunately without success.

Then I have tried also to add git submodule, but that drags whole git repository, while in certain cases I need only subset of whole git repository. What I have checked - it's indeed possible to perform sparse git checkout, but that require separate function, which I wrote below.

#-----------------------------------------------------------------------------
#
# Performs sparse (partial) git checkout
#
#   into ${checkoutDir} from ${url} of ${branch}
#
# List of folders and files to pull can be specified after that.
#-----------------------------------------------------------------------------
function (SparseGitCheckout checkoutDir url branch)
    if(EXISTS ${checkoutDir})
        return()
    endif()

    message("-------------------------------------------------------------------")
    message("sparse git checkout to ${checkoutDir}...")
    message("-------------------------------------------------------------------")

    file(MAKE_DIRECTORY ${checkoutDir})

    set(cmds "git init")
    set(cmds ${cmds} "git remote add -f origin --no-tags -t master ${url}")
    set(cmds ${cmds} "git config core.sparseCheckout true")

    # This command is executed via file WRITE
    # echo <file or folder> >> .git/info/sparse-checkout")

    set(cmds ${cmds} "git pull --depth=1 origin ${branch}")

    # message("In directory: ${checkoutDir}")

    foreach( cmd ${cmds})
        message("- ${cmd}")
        string(REPLACE " " ";" cmdList ${cmd})

        #message("Outfile: ${outFile}")
        #message("Final command: ${cmdList}")

        if(pull IN_LIST cmdList)
            string (REPLACE ";" "\n" FILES "${ARGN}")
            file(WRITE ${checkoutDir}/.git/info/sparse-checkout ${FILES} )
        endif()

        execute_process(
            COMMAND ${cmdList}
            WORKING_DIRECTORY ${checkoutDir}
            RESULT_VARIABLE ret
        )

        if(NOT ret EQUAL "0")
            message("error: previous command failed, see explanation above")
            file(REMOVE_RECURSE ${checkoutDir})
            break()
        endif()
    endforeach()

endfunction()


SparseGitCheckout(${CMAKE_BINARY_DIR}/catch_197 https://github.com/catchorg/Catch2.git v1.9.7 single_include)
SparseGitCheckout(${CMAKE_BINARY_DIR}/catch_master https://github.com/catchorg/Catch2.git master single_include)

I have added two function calls below just to illustrate how to use the function.

Someone might not like to checkout master / trunk, as that one might be broken - then it's always possible to specify specific tag.

Checkout will be performed only once, until you clear the cache folder.

Bibi

Answer 7

I would also recommend FetchContent for simple projects. However, sometimes more control is needed over the library being downloaded. For that ExternalProject is useful as it has many customization options. The secret is understanding the library you are trying to download. Read its CMakeLists.txt file to understand how you can modify installation directories, or what configuration is needed previous to build or installation.

The following example will download ZeroMQ (A.K.A. zmq) This library needs to run a bash script inside the library to configure things correctly and it has some options that can be used to customize your installation.

I personally don't like the standard locations where files are saved, as well as not needing to install the library in my system. So I want to install it within my project. Therefore you can specify the directories however you want.

In my project I override CMAKE_INSTALL_PREFIX and CMAKE_INCLUDE_PREFIX to point to the folders I desire the files to be.

set(ZMQ_INSTALL_DIR "${CMAKE_INSTALL_PREFIX}")
set(ZMQ_DIR "${CMAKE_CURRENT_BINARY_DIR}/ZeroMQ")
set(ZMQ_SRC_DIR "${ZMQ_DIR}/src")
set(ZMQ_INSTALL_INCLUDEDIR "${CMAKE_INCLUDE_PREFIX}/ZeroMQ")

include(ExternalProject)
ExternalProject_Add(ZeroMQ
  URL https://github.com/zeromq/libzmq/releases/download/v4.3.4/zeromq-4.3.4.tar.gz
  PREFIX "${ZMQ_DIR}/prefix"
  TMP_DIR "${ZMQ_DIR}/tmp"
  STAMP_DIR "${ZMQ_DIR}/stamp"
  DOWNLOAD_DIR "${ZMQ_DIR}"
  SOURCE_DIR "${ZMQ_SRC_DIR}"
  BINARY_DIR "${ZMQ_DIR}/build"
  CONFIGURE_COMMAND cd ${ZMQ_SRC_DIR} && ./configure --with-pic --prefix=${ZMQ_INSTALL_DIR}
  BUILD_COMMAND ""
  INSTALL_COMMAND cd ${ZMQ_SRC_DIR} && $(MAKE) install
)

Note that ZeroMQ is not the actual library target, it is a name I chose for the ExternalProject target. The library target would be zmq. So the library file would be libzmq.so.

After build or installation your main target needs to show it depends on the external library so that things are built in the right order. Therefor, you would need to add the following in the CMakeLists.txt where you target is (or in any parent CMakeLists.txt):

include_directories("${CMAKE_INCLUDE_PREFIX}/ZeroMQ")
add_dependencies(your_target ZeroMQ)     # Depends on the ExternalProject
target_link_libraries(your_target zmq)   # Link the library target

If you need to download another external library that depends on zmq you can add a DEPENDS argument to ExternalProject_Add:

ExternalProject_Add(
    cppzmqDownload
    URL https://github.com/zeromq/cppzmq/archive/refs/tags/v4.10.0.tar.gz
    DEPENDS ZeroMQ
    CMAKE_ARGS
      -DCMAKE_MODULE_PATH:PATH=${CMAKE_INSTALL_PREFIX}
      -DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_INSTALL_PREFIX}
      -DCMAKE_INSTALL_INCLUDEDIR:PATH=${CMAKE_INCLUDE_PREFIX}/cppzmq
      -DCPPZMQ_BUILD_TESTS=OFF
      -DPC_LIBZMQ_INCLUDE_DIRS=${ZMQ_INSTALL_INCLUDEDIR}
      -DPC_LIBZMQ_LIBRARY_DIRS=${CMAKE_LIB_PREFIX}
)

I hope this helps someone even though this is an old post.

Fluorene