Zeroth, an update
As of the next version of CMake (3.21), you may not want to use file(GET_RUNTIME_DEPENDENCIES) in some cases. (Which would be a good thing, as it works... poorly. It has no ability to differentiate between 32-bit and 64-bit shared libraries, for one thing, so it's irritatingly common to get wrong-arch libs returned on Linux. Then again, this development won't change that fact.)
If you're on Windows, the most common platform to require GET_RUNTIME_DEPENDENCIES logic, the next version of CMake is looking to take another stab at this (hopefully, fourth(?) time's the charm) with a new generator expression: $<TARGET_RUNTIME_DLLS:target>.
It's documented as the "List of DLLs that the target depends on at runtime. This is determined by the locations of all the SHARED and MODULE targets in the target's transitive dependencies. [...] This generator expression can be used to copy all of the DLLs that a target depends on into its output directory in a POST_BUILD custom command."
Considering I currently have custom logic in a CMakeLists.txt to do precisely that, because it's the only way to make the library's unit tests executable from the build directory, I'm hopeful this new expression makes that a bit easier.
Further update...
($<TARGET_RUNTIME_DLLS> won't fix the problems with file(GET_RUNTIME_DEPENDENCIES), but some commits just merged into CMake's upcoming 3.21 branch purport to, by teaching it how to distinguish between libraries for different architectures. Hooray!)
First, a caveat
You mentioned Qt. No matter what you do here, this method is unlikely to work for Qt all by itself, because there's no way using only the runtime dependencies of a program/library that you can discover any Qt plugins or other components that your installation may also require. Qt's dependencies are more complex than just libraries.
(My answer here demonstrates how to obtain Qt plugin information for bundling purposes, using the QCocoaIntegrationPlugin QPA on macOS as an example. All of Qt's plugins are represented by their own IMPORTED CMake targets, in recent releases, so it's typically possible to write install(CODE ...) scripting which picks up those targets using generator expressions in a similar manner to the following code.)
file(GET_RUNTIME_DEPENDENCIES)
As Tsyvarev noted in comments, GET_RUNTIME_DEPENDENCIES is intended to be used in the install stage, not the configure stage. As such, it needs to be placed in an install(CODE ...) or install(SCRIPT ...) statement, which will cause the code evaluation to be delayed until after the build is complete. (In fact, install(CODE ...) inserts the given code right into the current directory's cmake_install.cmake script. You can examine the results just by looking at that file, without even having to run the install.)
The delayed evaluation also comes with a few wrinkles. Primarily: The code doesn't understand targets. The targets no longer exist at the install stage. So, to include any target info, you have to use generator expressions to insert the correct values.
While the CMake documentation indicates that variable references and escapes aren't evaluated inside bracket arguments, generator expressions are. So, you can compose the CODE wrapped in [[ ]] to avoid escaping everything.
You still have to be careful about variable expansion / escaping. Most variables (including any you create) aren't available in the install context — only a few are, like CMAKE_INSTALL_PREFIX. You have to either expand or set any others.
There are, AFAICT, no generator expressions to access arbitrary variables. There are some for specific variables/values, but you can't say something like $<LIST:MY_LIST_VAR> or $<VALUE:MY_STRING_VAR> to combine variables and bracket arguments.
So, if you want to use variables from the configure context in the CODE, where they'll be evaluated at install time, the easiest thing to do is to "transfer" them into the install script by set()-ing a variable in the CODE.
file(INSTALL TYPE SHARED_LIBRARY)
To install shared library dependencies, you can use the same file(INSTALL) command that CMake itself uses in cmake_install.cmake if you build a shared library target. It uses the TYPE SHARED_LIBRARY option to add some extra processing. The FOLLOW_SYMLINK_CHAIN option is also especially handy. Together they'll make file(INSTALL) both resolve symbolic links in the source files, and automatically recreate them in the destination path.
Example code
So all in all, you'd want to do something like this:
set(MY_DEPENDENCY_PATHS /path/one /path/two)
# Transfer the value of ${MY_DEPENDENCY_PATHS} into the install script
install(CODE "set(MY_DEPENDENCY_PATHS \"${MY_DEPENDENCY_PATHS}\")")
install(CODE [[
file(GET_RUNTIME_DEPENDENCIES
LIBRARIES $<TARGET_FILE:mylibtarget>
EXECUTABLES $<TARGET_FILE:myprogtarget>
RESOLVED_DEPENDENCIES_VAR _r_deps
UNRESOLVED_DEPENDENCIES_VAR _u_deps
DIRECTORIES ${MY_DEPENDENCY_PATHS}
)
foreach(_file ${_r_deps})
file(INSTALL
DESTINATION "${CMAKE_INSTALL_PREFIX}/lib"
TYPE SHARED_LIBRARY
FOLLOW_SYMLINK_CHAIN
FILES "${_file}"
)
endforeach()
list(LENGTH _u_deps _u_length)
if("${_u_length}" GREATER 0)
message(WARNING "Unresolved dependencies detected!")
endif()
]])
* – (Note that using the DIRECTORIES argument on a non-Windows system will cause CMake to emit a warning, as files' dependencies are supposed to be resolvable using only the current environment.)
If the code gets too complex, there's always the option to create a separate script file copy_deps.cmake in the ${CMAKE_CURRENT_SOURCE_DIR} and use install(SCRIPT copy_deps.cmake). (A previous version of this answer suggested using file(GENERATE...) to build the script — that won't work, as the file isn't written until after processing the CMakeLists.txt.)
install(CODE)or withinstall(SCRIPT). And not withinstall(TARGETS)as you have tried. You may perform experiments with this command alone, in simple scenarios withoutinstall. – Packston