Best practice for dependencies on #defines?

Question

Is there a best practice for supporting dependencies on C/C++ preprocessor flags like -DCOMPILE_WITHOUT_FOO? Here's my problem:

> setenv COMPILE_WITHOUT_FOO
> make <Make system reads environment, sets -DCOMPILE_WITHOUT_FOO>
  <Compiles nothing, since no source file has changed>

What I would like to do is have all files that rely on #ifdef statements get recompiled:

> setenv COMPILE_WITHOUT_FOO
> make
  g++ FileWithIfdefFoo.cpp

What I do not want to is have to recompile everything if the value of COMPILE_WITHOUT_FOO has not changed.

I have a primitive Python script working (see below) that basically writes a header file FooDefines.h and then diffs it to see if anything is different. If it is, it replaces FooDefines.h and then the conventional source file dependency takes over. The define is not passed on the command line with -D. The disadvantage is that I now have to include FooDefines.h in any source file that uses the #ifdef, and also I have a new, dynamically generated header file for every #ifdef. If there's a tool to do this, or a way to avoid using the preprocessor, I'm all ears.

import os, sys
def makeDefineFile(filename, text):
    tmpDefineFile = "/tmp/%s%s"%(os.getenv("USER"),filename) #Use os.tempnam?
    existingDefineFile = filename

    output = open(tmpDefineFile,'w')
    output.write(text)
    output.close()

    status = os.system("diff -q %s %s"%(tmpDefineFile, existingDefineFile))

    def checkStatus(status):
        failed = False
        if os.WIFEXITED(status):
            #Check return code
            returnCode = os.WEXITSTATUS(status)
            failed = returnCode != 0
        else:
            #Caught a signal, coredump, etc.
            failed = True
        return failed,status

    #If we failed for any reason (file didn't exist, different, etc.)
    if checkStatus(status)[0]:
        #Copy our tmp into the new file
        status = os.system("cp %s %s"%(tmpDefineFile, existingDefineFile))
        failed,status  = checkStatus(status)
        print failed, status
        if failed:
            print "ERROR: Could not update define in makeDefine.py"
            sys.exit(status)

Fungicide

Answer 1

This is certainly not the nicest approach, but it would work:

find . -name '*cpp' -o -name '*h' -exec grep -l COMPILE_WITHOUT_FOO {} \; | xargs touch

That will look through your source code for the macro COMPILE_WITHOUT_FOO, and "touch" each file, which will update the timestamp. Then when you run make, those files will recompile.

If you have ack installed, you can simplify this command:

ack -l --cpp COMPILE_WITHOUT_FOO | xargs touch

Veriee

That's an interesting approach. I think the grep is sufficient because the dependencies tend to be flat, rather than depending on each other. Unfortunately for me, I have Perforce which leaves files in a read-only state unless they are checked out. If I were using git your solution would be perfect. – Fungicide

@Walter - I use Perforce too. You should be able to check out the entire directory, which would allow you to use this. But when you go to check in your changes, just remember to do "Revert Unchanged Files" first. – Veriee

BTW, it helps to know that ack is known as ack-grep on some systems. – Cockeye

Answer 2

I don't believe that it is possible to determine automagically. Preprocessor directives don't get compiled into anything. Generally speaking, I expect to do a full recompile if I depend on a define. DEBUG being a familiar example.

I don't think there is a right way to do it. If you can't do it the right way, then the dumbest way possible is probably the your best option. A text search for COMPILE_WITH_FOO and create dependencies that way. I would classify this as a shenanigan and if you are writing shared code I would recommend seeking pretty significant buy in from your coworkers.

CMake has some facilities that can make this easier. You would create a custom target to do this. You may trade problems here though, maintaining a list of files that depend on your symbol. Your text search could generate that file if it changed though. I've used similar techniques checking whether I needed to rebuild static data repositories based on wget timestamps.

Cheetah is another tool which may be useful.

If it were me, I think I'd do full rebuilds.

Skurnik

Unfortunately, I think you are right. On this team it would not be a problem to enforce use of a mechanism (especially since I built our make system [on top of QMake]). However, there isn't a standard or even preferred way to do this. Yet another reason the preprocessor should be used only when absolutely required. – Fungicide

Conditional compilation is always tedious. Using it sparingly with daily/integration builds are the least oppressive solution I've seen with this. The caveat being that most organizations aren't set up to do this on private branches. – Skurnik

Answer 3

Your problem seems tailor-made to treat it with autoconf and autoheader, writing the values of the variables into a config.h file. If that's not possible, consider reading the "-D" directives from a file and writing the flags into that file.

Under all circumstances, you have to avoid builds that depend on environment variables only. You have no way of telling when the environment changed. There is a definitive need to store the variables in a file, the cleanest way would be by autoconf, autoheader and a source and multiple build trees; the second-cleanest way by re-configure-ing for each switch of compile context; and the third-cleanest way a file containing all mutable compiler switches on which all objects dependant on these switches depend themselves.

When you choose to implement the third way, remember not to update this file unnecessarily, e.g. by constructing it in a temporary location and copying it conditionally on diff, and then make rules will be capable of conditionally rebuilding your files depending on flags.

Cockeye

Answer 4

One way to do this is to store each #define's previous value in a file, and use conditionals in your makefile to force update that file whenever the current value doesn't match the previous. Any files which depend on that macro would include the file as a dependency.

Here is an example. It will update file.o if either file.c changed or the variable COMPILE_WITHOUT_FOO is different from last time. It uses $(shell ) to compare the current value with the value stored in the file envvars/COMPILE_WITHOUT_FOO. If they are different, then it creates a command for that file which depends on force, which is always updated.

file.o: file.c envvars/COMPILE_WITHOUT_FOO
    gcc -DCOMPILE_WITHOUT_FOO=$(COMPILE_WITHOUT_FOO) $< -o $@

ifneq ($(strip $(shell cat envvars/COMPILE_WITHOUT_FOO 2> /dev/null)), $(strip $(COMPILE_WITHOUT_FOO)))
force: ;
envvars/COMPILE_WITHOUT_FOO: force
    echo "$(COMPILE_WITHOUT_FOO)" > envvars/COMPILE_WITHOUT_FOO
endif

If you want to support having macros undefined, you will need to use the ifdef or ifndef conditionals, and have some indication in the file that the value was undefined the last time it was run.

Greenfinch

Answer 5

make triggers on date time stamps on files. A dependent file being newer than what depends on it triggers it to recompile. You'll have to put your definition for each option in a separate .h file and ensure that those dependencies are represented in the makefile. Then if you change an option the files dependent on it would be recompiled automatically.

If it takes into account include files that include files you won't have to change the structure of the source. You could include a "BuildSettings.h" file that included all the individual settings files.

The only tough problem would be if you made it smart enough to parse the include guards. I've seen problems with compilation because of include file name collisions and order of include directory searches.

Now that you mention it I should check and see if my IDE is smart enough to automatically create those dependencies for me. Sounds like an excellent thing to add to an IDE.

Sclerodermatous

added a bug report for the IDE I use. With any luck they might fix it soon – Sclerodermatous

Answer 6

Jay pointed out that "make triggers on date time stamps on files".

Theoretically, you could have your main makefile, call it m1, include variables from a second makefile called m2. m2 would contain a list of all the preprocessor flags.

You could have a make rule for your program depend on m2 being up-to-date.

the rule for making m2 would be to import all the environment variables ( and thus the #include directives ).

the trick would be, the rule for making m2 would detect if there was a diff from the previous version. If so, it would enable a variable that would force a "make all" and/or make clean for the main target. otherwise, it would just update the timestamp on m2 and not trigger a full remake.

finally, the rule for the normal target (make all ) would source in the preprocessor directives from m2 and apply them as required.

this sounds easy/possible in theory, but in practice GNU Make is much harder to get this type of stuff to work. I'm sure it can be done though.

Gastroenterostomy

It's actually pretty easy. "include cppflagsfile" and "$(ENVDEPOBJS) : cppflagsfile" already do the trick. – Cockeye