How do I run a python script every time in a CMake build?

Question

Currently I have a need to run a Python script every time in CMake which generates a .qrc file. I can't use Qt Designer and I have to use CMake. set(CMAKE_AUTORCC ON) is being used and fails whenever a resource is added or a name is changed, thus the reason for the python script. The script itself already generates output and everything works after that, so I don't need output from CMake itself. I have currently tried the following:

include(FindPythonInterp)
set (py_cmd "QRC_Updater.py")
execute_process(
                  COMMAND ${PYTHON_EXECUTABLE} ${py_cmd}
                  WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}
                  RESULT_VARIABLE py_result
               )
message(STATUS "Python result: ${py_result})

and it works, but it does not execute every time. It only executes when CMakeLists.txt is modified.

So after some searching, people suggested using

add_custom_target(...)

and

add_custom_command(...)

which I have also tried with this:

add_custom_target(
   always_run_target ALL
   DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/__header.h
   )

add_custom_command(
    OUTPUT
        ${CMAKE_CURRENT_BINARY_DIR}/__header.h
        ${CMAKE_CURRENT_BINARY_DIR}/header.h
    COMMAND ${PYTHON_EXECUTABLE} ${py_cmd}
    )

which doesn't run the script at all. I know it's not running the script because I have the file open in Notepad++ and it doesn't ask if I want to switch to the newer version of the file like it does after execute_process() runs. Other variations of add_custom_command() also don't run the script. There are no errors associated with this running except when I don't include ${PYTHON_EXECUTABLE} which results in "%1 is not a valid Win32 command". So how can I effectively get a Python script to run every single time in CMake?

EDIT: The answers here do not work. How to always run command when building regardless of any dependency?

Haerr

Maybe you need WORKING_DIRECTORY in add_custom_command too? – Creel

I've actually used WORKING_DIRECTORY in the add_custom_command and ended with the same results. The script never runs. – Haerr

Possible duplicate of How to always run command when building regardless of any dependency? – Homozygous

This answer - https://mcmap.net/q/266963/-how-to-always-run-command-when-building-regardless-of-any-dependency - seems the most elegant way for achieve your purpose, not sure why it is not the best scored answer. – Homozygous

@Homozygous I tried that one as well. Script never runs. If you look in the comments it shows what I have already tried, almost verbatim. The OP of that question also says it doesn't work reliably. – Haerr

Sorry, but I see neither in the comments nor in the question post itself that you have tried add_custom_target without add_custom_command. – Homozygous

Fair enough. I have tried add_custom_command to make it look almost like execute process. – Haerr

I have also used add_custom_target by itself. – Haerr

add_custom_target should definitely work - every run of make it should execute the COMMAND specified. It seems that your "not working" means something different. – Homozygous

You are correct. Add custom target should work. However, the script is not being run. I have tried multiple variations, all of which were found on this site, to no avail. The most I got out of add_custom_target was it creating a new project, which is not what I want. Even then the script still didn't run. – Haerr

Well, as something which "should work" doesn't work for you, it is signal to tell us more about your environment. I see you are on Windows. Which CMake generator do you use (Visual Studio, NMake, MinGW)? How do you run the build process (a button in IDE, make, cmake --build)? You said that add_custom_target creates a new project - in Visual Studio every target is actually a new project. – Homozygous

The machine this is on is a Windows 7 machine. The CMake GUI version 3.7.2, but it uses Visual Studio 14 2015 as the generator. A build command is sent from Visual Studio 2015 that calls CMake to build. I don't need a new project as the project this script needs to run for already exists. – Haerr

Hm, as far as I may guess, in Visual Studio you actually build some specific target, not a default one. Because of that custom target is not executed. For make it work, make the target which you build dependent from the always-run target: add_dependency(<your-target> always_run_target). – Homozygous

Answer 1

you need to add a dependency to custom command to inspect changes on OriginalHeader.h and regenerates __header.h and header.h if it changes.

add_executable(MyExe main.cpp ${CMAKE_CURRENT_BINARY_DIR}/__header.h)

add_custom_target(
        always_run_target ALL
        DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/__header.h
)

add_custom_command(
        OUTPUT
            ${CMAKE_CURRENT_BINARY_DIR}/__header.h
            ${CMAKE_CURRENT_BINARY_DIR}/header.h
        DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/OriginalHeader.h
        COMMAND python ${py_cmd}
)

I only changed the ${PYTHON_EXECUTABLE} with python. I was able to run this python script (below) that print, create a directory and generate two files __header.h and header.h every time a change is detected in OriginalHeader.h, or __header.h/header.h don't exists yet.

import os

print("TEST")

if not os.path.exists("TESTDIR"):
    os.makedirs("TESTDIR")
with open("header.h", 'w+'):
    os.utime("header.h", None)

Tyrocidine

I don't need an executable file created from this, I just need my Python script to run every time. __header.h is a fake file to trick CMake into running every time, and header.h is a file I don't care about because the Python script creates everything I need. I apologize for not being specific about that. I can't rely on a change being detected because people aren't supposed to change the .qrc file directly. It needs to be created by the script whenever the project is built to account for additions, removals, and name changes. – Haerr

Okay so why custom command should output __header.h ? it should depend on it ? no – Tyrocidine

From what I see now, if __header.h and header.h, don't exists and won't the first time the project is built, it should call the python script. Adding the depends in custom_command should trigger, the regeneration of __header.h and header.h, and so a new call to the python script. – Tyrocidine

It's part of the example I used to trick CMake into running that command every time. If you look at the link I provided and go to ideasman42's answer it explains. – Haerr

Do you think your script is okay ? Because, for me the above code works, every time I rebuild a project with or without DEPENDS Keyword on the custom command. I'm using CMake 3.3.2 – Tyrocidine

My script is definitely OK because I got that working first before I even started trying to incorporate it into CMake. I have execute_process() working just fine, it just doesn't perform every time. – Haerr

Okay it seems weird, could you try my script instead of yours with my code and tell me when you rebuild if the script is executed at least once ? – Tyrocidine

The script isn't executed until that new target's project is called to build explicitly and doesn't execute when the project that needs it is called to build. The project is built fine otherwise. – Haerr

So, tell me if I'm right, your use case is : - people add a resource file to a resource folder, or rename one for a target - then he want to build with latest resource - run build with CMake Command (or generator) - scripts is running, generate qrc file with the newest resources. - project's target depends on qrcfile and so is building. - project is running and works fine with the new resources. – Tyrocidine

Additionally I have few questions : Which version of CMake do you use ? Which command do you use when building the project ? Which cmake's generator do you use ? Do you clean the project's cache before calling build again ? If not it might not need to rebuild since nothing has changed. – Tyrocidine

No, that is not correct. I think you're confusing resource with Qt resource, which is something different. The machine this is on is a Windows 7 machine. The CMake GUI version 3.7.2, but it uses Visual Studio 14 2015 as the generator. A build command is sent from Visual Studio 2015 that calls CMake to build. Cleaning the cache does nothing. – Haerr

Answer 2

I have a solution if you are always building ALL targets, and for individual targets.

To run Python script when building all

I have a separate generate_header.cmake file and added it to the root-level directory of the project, and the file contains the following:

# File: generate_header.cmake
#############################

find_package(PythonInterp)
find_package(Python)

function(generate_headers target_name)
    if (PYTHON_FOUND)
        add_custom_target(${target_name} ALL
            COMMAND 
               ${PYTHON_EXECUTABLE} ${py_cmd}
            OUTPUT 
               ${CMAKE_CURRENT_BINARY_DIR}/__header.h
               ${CMAKE_CURRENT_BINARY_DIR}/header.h  
        )
    endif()
endfunction()

and in the root-level CMakeLists.txt I have added the following:

include(generate_header)

# ... other project stuff

generate_headers(always_run_target)

Now every time I build all, i.e. pass --target all when calling cmake.exe, my script runs.

Note: Building other individual targets does not regenerate the header files.

To run Python script when building individual targets

If you want to generate for individual targets, then you need to call add_custom_command for each target. Note that this will also get called when building all, so it will call the script multiple times if you have multiple targets:

add_custom_command(
    TARGET name_of_your_target
    PRE_BUILD # Call this command pre-build
    COMMAND ${PYTHON_EXECUTABLE} ${py_cmd}
    COMMENT "Generate Headers"
)    

Americana