What are the (dis)advantages of writing unit tests in a different language to the code?

Question

Unit tests have different requirements than production code. For example, unit tests may not have to be as performant as the production code.

Perhaps it sometimes makes sense to write your unit tests in a language that is better suited to writing unit tests? The specific example I have in mind is writing an application in C# but using IronRuby or IronPython to write the tests.

As I see it, using IronPython and IronRuby have several advantages over C# code as a testing language:

• Mocking can be simpler in dynamically typed languages
• IronPython has less verbose type annotations that are not needed in unit tests
• Experimental invocation of tests without recompilation by typing commands at the interpreter

What are the tradeoffs in using two different languages for tests and production code?

Sacellum

Answer 1

Disadvantages that come to my mind:

• Depending on the language, you need another development environment (additional dependency of the project, additional effort to setup a development machine, additional licenses, additional training ...)
• Refactoring is sometimes supported by the IDE - but most probably not for this other language. So you have to refactor them manually.
• Unit tests can also be used as programming examples. Tests show how the tested classes are intended to be used. This does not work so well if the tests are written in a different language.

Coco

+1 for tests as programming examples; it's also a great way to see how usable your code is. On the other hand, you could argue that writing a test suite in VB.Net against a C# API could test the usability of that API for that other language. – Reflexive

Answer 2

One obvious potential problem is a confusing debugging experience. I haven't personally tried to debug across languages in .NET - what happens if you step into C# code from IronPython?

The other problem is that anyone wanting to develop on your code base has to know both languages.

Linzer

This works pretty transparently with C# and F# in VS2010; the debugger moves between the source just as you'd expect. I can't speak for other languages. – Unseasonable

I maintained a VB.net/C# mixed solution - debugger worked fine, but your head isn't always so flexible. – Cornute

Answer 3

I'm a huge fan of the Iron languages but there are significant disadvantages in using them to test statically compiled code. Testing Ruby/Python with Ruby/Python works great, i.e., you can fake objects in any number of ways. But testing C# with Ruby/Python can lead to more complexity than may be desired.

Consider a scenario where you are writing an ASP MVC project where you would like to fake out the type of browser making a request to the controller. In C#, you could mock it out like this (using Moq):

var fakeRequest = new Moq.Mock<System.Web.HttpRequestBase>();
fakeRequest.Setup(request => request.Browser.Type).Returns("Fake");

In IronRuby, it would look something like this:

class FakeBrowser < HttpBrowserCapabilitiesBase
  def type
    "Fake"
  end
end

class FakeRequest < HttpRequestBase
  def browser
    FakeBrowser.new
  end
end

The deeper the context that needs to be faked out, the more complicated it gets. In C#, the mocking framework does the subtyping automatically for you but in a dynamic language you will, unfortunately, have a lot of work ahead of you. Mocking libraries like PyMock or Mocha will not work for tests like this because the code under test has strong typing dependencies and the fake objects generated by Ruby/Python frameworks will not conform to those dependencies. I would love to see the situation improve as both IronRuby and IronPython mature but right now the testing C# story isn't so good.

If my take on this is wrong, I'd love to stand corrected. =)

Torture

I think this example is a little bit unfair: you are comparing code using a mocking framework versus an open-coded mock. I'm pretty sure, if you used the same approach in both cases (i.e. either don't use Moq in the C# case or use Moq in the Ruby case), the difference would be far less dramatic. – Selfdriven

That's a fair comment but as far as I know, libraries that depend on Linq Expressions are unusable* in the Iron languages. Again, I'd love to be proven wrong. * rubyforge.org/pipermail/ironruby-core/2009-May/004536.html – Torture

Answer 4

Main disadvantage as I see it is maintainablity. If you code in C#, your development team are competent in that language, as will new hires be. You need a multi-functional dev team.

I think that it's also worth noting that you probably don't want people writing their tests in a language that is maybe not their strongest. Your test code needs to be robust.

You also need to be switching between syntaxes whilst writing codes/tests - this is a bit of a nuisance.

Cornute

Answer 5

I think that this is an excellent question and an idea worthy of consideration - particularly in an environment like Visual Studio/.NET where this is easily supported

The plus side - as you suggest - is that you can choose to use a language/tool to create tests that is more suited to creating tests than perhaps the code you are using to create code and for this reason alone its worth a thought.

The down side is, as suggested, that your developers - those creating the tests (and we must remember not to confuse Unit Testing with Test Driven Design) probably ought to be fluent in more than one language (I'd suggest that the ability to be so is fairly important to a good developer but I'm biased!) - and more importantly that you may have to worry about structural differences between the two (though again, if you're talking about .NET languages that should be covered for you).

It gets even more interesting if you go beyond "unit" tests to tests at all levels where the specific capabilities of particular languages may give advantages in building up a test case.

The ultimate question is whether the advantages outweigh the disadvantages... and that's probably somewhat case specific.

Remissible

Answer 6

The biggest tradeoff would be if someone was looking at using Unit Tests to figure out how a certain action may be performed, writing it in a different language would make it harder to use. Also you would have to make your C# code CLS compliant.

Sello

Answer 7

When building an API or library, I often deliver unit tests as a form of documentation on how best to use the API or library. Most of the time I build a C# library, I'm delivering to a client who will be writing C# code to use the library.

For documentation sake, at least some of my tests will always be written in the target language.

Paredes

Answer 8

The biggest disadvantage is that you are also testing compiler dependencies in your unit tests. In a sense, this also makes them integration tests. That might make them preferable if you expect your code to be usable from multiple languages, but it's adding one level of complexity that you may not need if your code will only be used in production with the language that it's developed in.

One advantage that I can see is that it further isolates that code being developed from the test itself. By separating the act of writing the test even further from the actual code under development, it forces you to really consider how the code should be written to pass the test rather than simply moving the initial development of the code into the test itself.

Sansen

Answer 9

I agree with the other answers that there are disadvantages, but I'm surprised that so few have talked about advantages.

• TDD'ing across languages is a great way to learn new languages: write the tests in a language you know well, and the implementation in the language you are learning. As you are learning, you will discover better ways of writing the code than you first did, but refactoring is easy because you have a test suite.
• Having to master multiple languages keeps you (and your team) sharp.
• You get better verification that your API is interoperable across languages.

Auvil

Answer 10

Experience 1

In the past year I worked on a project that had two main pieces:

1. A Grails web application
2. and a Java Swing application

we wrote our Java unit tests using Groovy and it worked out well. Unit testing with groovy took a lot less time verbosity wise and also made it possible to fake static methods and so forth. Although there were a couple places where we ran into unexpected results due to Groovy's dynamic typing, on the whole it was a positive experience.

Experience 2

Another recent project I worked on was a C# ASP.NET MVC 3 web application where I wrote all the unit tests with F#. I chose C# for the web app because I felt it worked better with MVC 3. I chose F# for unit tests because it is my favorite language. The only issues I ran into were minor annoyances with types like null vs. option.

Conclusion

When we have two languages targeting the same runtime (well, C# / F# and Java / Groovy on CLR and JRE respectively) where one is used for production code and one for unit tests, there isn't too much to worry about regarding compatibility at least. Then I think it's really a questions of whether you and your team feel comfortable enough in both languages (and I suppose you might be kind enough to consider future maintainers as well). Indeed, I think the times you'd be compelled to use a different language for unit testing is when the unit testing language is actually your language of comfort or choice over your production language.

There are some exceptions I'd admit to this liberal attitude. If you're are designing a library to be consumed by language X then it may be a smart idea to write your unit tests in language X (some at least). But for application code, I've never found writing unit tests in the same language as your production code particularly advantageous.

Ephesian