Writing unit tests in my compiler (which generates IL)

Question

I'm writing a Tiger compiler in C# and I'm going to translate the Tiger code into IL.

While implementing the semantic check of every node in my AST, I created lots of unit tests for this. That is pretty simple, because my CheckSemantic method looks like this:

public override void CheckSemantics(Scope scope, IList<Error> errors) {
...
}

so, if I want to write some unit test for the semantic check of some node, all I have to do is build an AST, and call that method. Then I can do something like:

Assert.That(errors.Count == 0);

or

Assert.That(errors.Count == 1);
Assert.That(errors[0] is UnexpectedTypeError);
Assert.That(scope.ExistsType("some_declared_type"));

but I'm starting the code generation in this moment, and I don't know what could be a good practice when writing unit tests for that phase.

I'm using the ILGenerator class. I've thought about the following:

1. Generate the code of the sample program I want to test
2. Save generated code as test.exe
3. Execute text.exe and store the output in results
4. Assert against results

but I'm wondering if there is a better way of doing it?

Keratin

Answer 1

That's exactly what we do on the C# compiler team to test our IL generator.

We also run the generated executable through ILDASM and verify that the IL is produced as expected, and run it through PEVERIFY to ensure that we're generating verifiable code. (Except of course in those cases where we are deliberately generating unverifiable code.)

Shelburne

It is good to know that. I'll do it that way then. I was just a little bit worried about the performance of lots of tests running, and creating, reading and deleting file(s) on the disk. – Keratin

@OscarMederos: If the performance is not good enough then either (1) do profiling to figure out what is slow and fix it if you can, or (2) change your test strategy so that some tests run on every checkin, some tests run overnight, and some run over the weekend. That way you get a good balance of rapidly discovering problems and still doing thorough testing. – Shelburne

Answer 2

I've created a post-compiler in C# and I used this approach to test the mutated CIL:

1. Save the assembly in a temp file, that will be deleted after I'm done with it.
2. Use PEVerify to check the assembly; if there's a problem I copy it to a known place for further error analysis.
3. Test the assembly contents. In my case I'm mostly loading the assembly dynamically in a separate AppDomain (so I can tear it down later) and exercising a class in there (so it's like a self-checking assembly: here's a sample implementation).

I've also given some ideas on how to scale integration tests in this answer.

Rental

Answer 3

You can think of testing as doing two things:

1. letting you know if the output has changed
2. letting you know if the output is incorrect

Determining if something has changed is often considerably faster than determining if something is incorrect, so it can be a good strategy to run change-detecting tests more frequently than incorrectness-detecting tests.

In your case you don't need to run the executables produced by your compiler every time if you can quickly determine that the executable has not changed since a known good (or assumed good) copy of the same executable was produced.

You typically need to do a small amount of manipulation on the output that you're testing to eliminate differences that are expected (for example setting embedded dates to a fixed value), but once you have that done, change-detecting tests are easy to write because the validation is basically a file comparison: Is the output the same as the last known good output? Yes: Pass, No: Fail.

So the point is that if you see performance problems with running the executables produced by your compiler and detecting changes in the output of those programs, you can choose to run tests that detect changes a stage earlier by comparing the executables themselves.

Tenant