Implementing `make check` or `make test`

Question

How can I implement a simple regression test framework with Make? (I’m using GNU Make, if that matters.)

My current makefile looks something like this (edited for simplicity):

OBJS = jscheme.o utility.o model.o read.o eval.o print.o

%.o : %.c jscheme.h
    gcc -c -o $@ $<

jscheme : $(OBJS)
    gcc -o $@ $(OBJS)

.PHONY : clean

clean :
    -rm -f jscheme $(OBJS)

I’d like to have a set of regression tests, e.g., expr.in testing a “good” expression & unrecognized.in testing a “bad” one, with expr.cmp & unrecognized.cmp being the expected output for each. Manual testing would look like this:

$ jscheme < expr.in > expr.out 2>&1
$ jscheme < unrecognized.in > unrecognized.out 2>&1
$ diff -q expr.out expr.cmp # identical
$ diff -q unrecognized.out unrecognized.cmp
Files unrecognized.out and unrecognized.cmp differ

I thought to add a set of rules to the makefile looking something like this:

TESTS = expr.test unrecognized.test

.PHONY test $(TESTS)

test : $(TESTS)

%.test : jscheme %.in %.cmp
    jscheme < [something.in] > [something.out] 2>&1
    diff -q [something.out] [something.cmp]

My questions:
• What do I put in the [something] placeholders?
• Is there a way to replace the message from diff with a message saying, “Test expr failed”?

Lithe

Whence all the tempfiles? What is wrong with – Enthymeme

@reinierpost: If you have a better way of doing these comparisons, by all means post an answer that includes them—that’s exactly the kind of help I’m asking for. – Lithe

Sorry, I thought I canceled that question. You do need tempfiles. – Enthymeme

Answer 1

Your original approach, as stated in the question, is best. Each of your tests is in the form of a pair of expected inputs and outputs. Make is quite capable of iterating through these and running the tests; there is no need to use a shell for loop. In fact, by doing this you are losing the opportunity to run your tests in parallel, and are creating extra work for yourself in order to clean up temp files (which are not needed).

Here's a solution (using bc as an example):

SHELL := /bin/bash

all-tests := $(addsuffix .test, $(basename $(wildcard *.test-in)))

.PHONY : test all %.test

BC := /usr/bin/bc

test : $(all-tests)

%.test : %.test-in %.test-cmp $(BC)
    @$(BC) <$< 2>&1 | diff -q $(word 2, $?) - >/dev/null || \
    (echo "Test $@ failed" && exit 1)

all : test 
    @echo "Success, all tests passed."

The solution directly addresses your original questions:

• The placeholders you're looking for are $< and $(word 2, $?) corresponding to the prerequisites %.test-in and %.test-cmp respectively. Contrary to the @reinierpost comment temp files are not needed.
• The diff message is hidden and replaced using echo.
• The makefile should be invoked with make -k to run all the tests regardless of whether an individual test fails or succeeds.
• make -k all will only run if all the tests succeed.

We avoid enumerating each test manually when defining the all-tests variable by leveraging the file naming convention (*.test-in) and the GNU make functions for file names. As a bonus this means the solution scales to tens of thousands of tests out of the box, as the length of variables is unlimited in GNU make. This is better than the shell based solution which will fall over once you hit the operating system command line limit.

Athenian

Could you include a link to the relevant GNU Make manual entry at ‹https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html›? – Lithe

Thanks, I learned quite a few things by pulling out the GNU make manual and working through this example! – Hamper

Answer 2

Make a test runner script that takes a test name and infers the input filename, output filename and smaple data from that:

#!/bin/sh
set -e
jscheme < $1.in > $1.out 2>&1
diff -q $1.out $1.cmp

Then, in your Makefile:

TESTS := expr unrecognised

.PHONY: test
test:
    for test in $(TESTS); do bash test-runner.sh $$test || exit 1; done

You could also try implementing something like automake's simple test framework.

Tache

I like the shell loop idea over my original plan of separate targets. The shell script’s not a bad idea, but I think I’ll incorporate the whole thing into the makefile. – Lithe

Also I’m really not looking to open up the whole autotools can of worms. – Lithe

@jcsalomon: I didn't mean use the autotools, but rather study how the testing system behaves. I'll leave this here in case you change your mind: lrde.epita.fr/~adl/autotools.html . Remember to escape the $ signs in your Makefile. It's caught me a couple of times. – Tache

Answer 3

What I ended up with looks like this:

TESTS = whitespace list boolean character \
    literal fixnum string symbol quote

.PHONY: clean test

test: $(JSCHEME)
    for t in $(TESTS); do \
        $(JSCHEME) < test/$$t.ss > test/$$t.out 2>&1; \
        diff test/$$t.out test/$$t.cmp > /dev/null || \
            echo Test $$t failed >&2; \
    done

It’s based on Jack Kelly’s idea, with Jonathan Leffler’s tip included.

Lithe

You'll want there to be a || exit 1 somewhere in there; otherwise you're simply ignoring all test failures. – Kunz

@l0b0, I don’t want to bail out after the first failure either. Tell you what: you come up with a clean way to exit 1 if any of the test fail—but after all tests run, and I’ll change my accepted answer to that one. – Lithe

@ChandanChoudhury, your question deserves its own post; the comments section is not a useful place to try to diagnose your Makefile’s problem. – Lithe

Answer 4

I'll address just your question about diff. You can do:

diff file1 file2 > /dev/null || echo Test blah blah failed >&2

although you might want to use cmp instead of diff.

On another note, you might find it helpful to go ahead and take the plunge and use automake. Your Makefile.am (in its entirety) will look like:

bin_PROGRAMS = jscheme
jscheme_SOURCES = jscheme.c utility.c model.c read.c eval.c print.c jscheme.h
TESTS = test-script

and you will get a whole lot of really nice targets for free, including a pretty full-featured test framework.

Twosome