How do I use CMake ExternalProject_Add or alternatives in a cross-platform way?

Question

I would like to build a third-party project that already has CMake as part of my project's CMake strips. ExternalProject_Add is for this purpose, but I have found it can only be made to work with a specific generator, and I wanted it to work on many platforms easily.

For example, here is my external project with an added script for zlib, which has its own CMakeLists.txt:

set(USE_PROJECT_CMAKE_MODULE_PATH "-DCMAKE_MODULE_PATH=${MAKE_MODULE_PATH}")
ExternalProject_Add(ZLIB
                    SOURCE_DIR ${CMAKE_SOURCE_DIR}/external/zlib
                    DOWNLOAD_COMMAND ""
                    UPDATE_COMMAND ""
                    CMAKE_ARGS
                       -DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
                       -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS}
                       -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS}
                       -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
                       ${USE_PROJECT_CMAKE_MODULE_PATH}
                    INSTALL_COMMAND "")

ExternalProject_Add_Step(ZLIB installInternally
                         COMMAND cd <BINARY_DIR> && make install
                         DEPENDEES install
                         ALWAYS 1)
ExternalProject_Get_Property(ZLIB install_dir)

if(UNIX)
    set(ZLIB_NAME libz)
else(UNIX)
    set(ZLIB_NAME zlib)
endif(UNIX)

add_library(zlib UNKNOWN IMPORTED)
set_property(TARGET zlib PROPERTY IMPORTED_LOCATION ${install_dir}/lib/${ZLIB_NAME}.a)
set(ZLIB_LIBRARIES zlib)
set(ZLIB_LIBRARIES_OPTIONAL ${ZLIB_LIBRARIES})
set(ZLIB_DIR ${install_dir} CACHE INTERNAL "zlib ROOT dir")
set(ZLIB_INCLUDE_DIRS ${install_dir}/include CACHE INTERNAL "zlib include dirs")
set(ZLIB_DEFINES "-msse2 -mfpmath=sse" CACHE INTERNAL "zlib defines")

The problem with this is that it works with make, but not with Xcode or Visual Studio. Perhaps there is some way to take the CMake build commands passed to my project and forward them to ExternalProject_Add.

How can I write ExternalProject_Add calls in a cross-platform way with minimal code complexity, or is there a better alternative?

Interlude

Found another tool that can serve this purpose: conan.io – Interlude

Answer 1

Problems

-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}

This is enough for single-configuration projects. But for Xcode and Visual Studio, you need to set CMAKE_CONFIGURATION_TYPES plus call build . --config at the build stage. See my answer.

COMMAND cd <BINARY_DIR> && make install

This will work only for Makefile generators of course. To be cross-platform you can use:

--build . --target install --config inside INSTALL_COMMAND of ExternalProject_Add.

Take a look at this template file, and in particular the following lines:

ExternalProject_Add(
    "${current_project}"
    URL
    @HUNTER_PACKAGE_URL@
    URL_HASH
    SHA1=@HUNTER_PACKAGE_SHA1@
    DOWNLOAD_DIR
    "@HUNTER_PACKAGE_DOWNLOAD_DIR@"
    SOURCE_DIR
    "@HUNTER_PACKAGE_SOURCE_DIR@"
    INSTALL_DIR
    "@HUNTER_PACKAGE_INSTALL_PREFIX@"
        # Not used, just avoid creating Install/<name> empty directory
    BUILD_COMMAND ""
        # This command is empty because all necessary targets will
        # be built on install stage
    CMAKE_ARGS
    "-G@CMAKE_GENERATOR@"
    "-C@HUNTER_CACHE_FILE@"
    "-C@HUNTER_ARGS_FILE@"
    "-D${postfix_name}=${${postfix_name}}"
    "-DCMAKE_BUILD_TYPE=${configuration}"
    "-DCMAKE_CONFIGURATION_TYPES=${configuration}"
    "-DCMAKE_INSTALL_PREFIX=@HUNTER_PACKAGE_INSTALL_PREFIX@"
    "-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}"
    INSTALL_COMMAND
        "@CMAKE_COMMAND@"
        --build .
        --target install
        --config ${configuration}
        --
        ${jobs_option}
)

Alternative

or is there a better alternative?

Have you seen Hunter?

You can add zlib just like this:

hunter_add_package(ZLIB)
find_package(ZLIB CONFIG REQUIRED)
target_link_libraries(... ZLIB::zlib)

This code works everywhere. Third party dependencies will be downloaded automatically in the configuration step. Example of building with different generator/toolchains (build.py is just a CMake wrapper that sets CMAKE_TOOLCHAIN_FILE and -G/-B):

build.py --toolchain mingw --config Release # MinGW Makefiles
build.py --toolchain vs-12-2013 --config Debug # Visual Studio 12 2013
build.py --toolchain xcode --config Release # Xcode
build.py --toolchain libcxx --config Release # Makefile with -stdlib=libc++ toolchain
build.py --toolchain ios-8-2 --config Release # Xcode with iOS SDK 8.2 toolchain

You got full control what options, build types or number of jobs you want to have while building third-party packages. For instance, this is how you can build four types, Debug, Release, MinSizeRel, and RelWithDebInfo for zlib and link MinSizeRel to the current project:

> build.py --toolchain xcode --verbose --config MinSizeRel --fwd "HUNTER_CONFIGURATION_TYPES=Release;Debug;MinSizeRel;RelWithDebInfo"
/.../clang  /.../lib/libz-MinSizeRel.a ... -o /.../_builds/xcode/MinSizeRel/foo

> ls -la /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz*
   99056 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz-MinSizeRel.a
  307872 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz-RelWithDebInfo.a
  109536 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libz.a
  258904 /.../.hunter/_Base/d1232c0/326318e/37e4682/Install/lib/libzd.a

Briton

Cool, that "--build ..." is useful, could you explain exactly what it is doing? Why is the build command empty? I'm also interested in hunter so I created a github issue with a few questions over there. – Interlude

@AndrewHundt could you explain exactly what it is doing several things like setting generator -G, build directory (-B_builds/xcode for xcode toolchain or -B_builds/vs-12-2013 for vs-12-2013 toolchain) or some environment variables. For example mingw --config Release is equal to: set PATH=%MINGW_PATH%;%PATH% + cmake -H. -B_builds/mingw-Release "-GMinGW Makefiles" -DCMAKE_BUILD_TYPE=Release – Briton

Why is the build command empty? you mean why there is no build.py --configure step? Script checks that there is no CMakeCache.txt file and run CMake configure. Since CMake detects code changes automatically further you can just run cmake --build. Anyway build.py is not mandatory for Hunter, it's just good aliasing tool I find very useful. – Briton

Sorry my question was very unclear I meant to ask why this build command was empty. Thanks for explaining the build.py, if it is simplifying some command line parameters and that's all it is doing that is actually a good thing! – Interlude

@AndrewHundt I meant to ask why this build command was empty Okay, I see. I've added some comments – Briton

Oh I see, it is redundant because you subsequently use the @CMAKE_COMMAND@... which does all of these steps for you – Interlude

I would be very interested to know what the @ variables have as values, but unfortunately I am not familiar with Hunter. In particular, @HUNTER_PACKAGE_INSTALL_PREFIX@ – Epact

What if we want to force a specific build configuration. For example if external projects are to be build as release regardless of the build configuration of the main project? – Mariner

Answer 2

CMake ExternalProject_Add calls work cross-platform by default and will only fail to do so if one uses particular commands that are only available on a subset of operating systems.

Typically, CMAKE_ARGS is used to pass information to each superbuild unit within an external project build. The CMakeLists.txt files that control each miniature part of the overall build use CMake's declarative syntax (e.g., "add_library(library_name SHARED filename1.hpp filename1.cpp). CMake will convert such syntax to the commands that are specific to the particular build system you wish to use (e.g., make and Ninja).

The sample above re: zlib fails to be cross-platform in part because the ExternalProject_Add_Step contains "COMMAND cd && make install", which necessarily only works in situations where invoking "cd" is actually the correct way to change directories, and where invoking "make" is actually the correct way to build software.

CMake's -E option provides a way to invoke basic operations like changing/copying/making/removing directories without making such assumptions.

(By the way, if you're using IDEs such as Visual Studio or Xcode, you'll likely want to invoke one or more IDE generators when using CMake. For instance, setting

-G "Eclipse CDT4 - Unix Makefiles" -DCMAKE_ECLIPSE_GENERATE_SOURCE_PROJECT=TRUE

will cause Eclipse projects to be generated in each build area, and also in the source code area that is shared for all builds. Of course, if you are using Xcode or Visual Studio, you'll have to substitute the appropriate flag for those IDEs. Alternatively, you could consider using Eclipse with Ninja on all platforms, though at the time of writing, I am not completely certain that Ninja is ready for prime-time on non-Linux, non-Windows operating systems.)

Dromond

Thanks for attempting a reply, but I think your reply is actually unrelated to the question. I'm not asking about zlib, it is only an example. I'm specifically asking in general about using the cmake function ExternalProject_Add so that it can work across platforms so I don't need to instantiate many versions of that call for each platform and configuration. – Interlude

Thanks for pointing out that I misunderstood your question. I have revised my post to attempt to answer what you are really after. – Dromond

I see, so I'm trying to see if I could replace cd <BINARY_DIR> && make install with cmake -E build <BINARY_DIR> but I don't think that will do the install step that I actually want. Also, I still have the issue where Xcode, for example, puts files in buildDir/bin/Debug/binary and buildDir/bin/Release/binary, where make puts things in buildDir/bin/binary. Side rant: It is a bit frustrating to have to account for all these platform specific differences, and I'm hoping CMake has a cleaner way to handle it that I'm missing at the moment. – Interlude