Can automated unit testing replace static type checking?

Question

I've started to look into the whole unit testing/test-driven development idea, and the more I think about it, the more it seems to fill a similar role to static type checking. Both techniques provide a compile-time, rapid-response check for certain kinds of errors in your program. However, correct me if I'm wrong, but it seems that a unit test suite with full coverage would test everything static type checking would test, and then some. Or phrased another way, static type checks only go part of the way to "prove" that your program is correct, whereas unit tests will let you "prove" as much as you want (to a certain extent).

So, is there any reason to use a language with static type checking if you're using unit testing as well? A somewhat similar question was asked here, but I'd like to get into more detail. What specific advantages, if any, does static type checking have over unit tests? A few issues like compiler optimizations and intellisense come to mind, but are there other solutions for those problems? Are there other advantages/disadvantages I haven't thought of?

Manor

Answer 1

I would think that automated unit testing will be important to dynamic typed languages, but that doesn't mean it would replace static type checking in the context that you apply. In fact, some of those who use dynamic typing might actually be using it because they do not want the hassles of constant type safety checks.

The advantages dynamically typed languages offer over static typed languages go far beyond testing, and type safety is merely one aspect. Programming styles and design differences over dynamic and static typed languages also vary greatly.

Besides, unit tests that are written too vigorously enforce type safety would mean that the software shouldn't be dynamically typed after all, or the design being applied should be written in a statically typed language, not a dynamic one.

Degenerate

Answer 2

There is one immutable fact about software quality.

If it can't compile, it can't ship

In this rule, statically typed languages will win over dynamically typed languages.

Ok, yes this rule is not immutable. Web Apps can ship without compiling (I've deployed many test web apps that didn't compile). But what is fundamentally true is

The sooner you catch an error, the cheaper it is to fix

A statically typed language will prevent real errors from happening at one of the earliest possible moments in the software development cycle. A dynamic language will not. Unit Testing, if you are thorough to a super human level can take the place of a statically typed language.

However why bother? There are a lot of incredibly smart people out there writing an entire error checking system for you in the form of a Compiler. If you're concerned about getting errors sooner use a statically typed language.

Please do not take this post as a bashing of dynamic languages. I use dynamic languages daily and love them. They are incredibly expressive and flexible and allow for incredibly fanscinating program.s However in the case of early error reporting they do lose to statically typed languages.

Brawn

My only problem with your post is that, "if it can compile" doesn't necessarily mean it can ship either. – Degenerate

No it certainly can't. The point of the post though is that compilation is barrier #1. There should be many other barriers in a software project for instance static analysis (FxCop), unit tests, QA tests, smoke tests, app building ... – Brawn

"It compiles, ship it!" That has been a joke since before 2009. I'm still waiting for a single example of a bug (an actual bug: crashing program, wrong results/behavior, vulnerability) that static typing can find that unit tests CANNOT find. Take a classic example: off-by-one. How do you define your C++/C#/Java type system so it'll find these (all of them)? Unit tests surely can find those. – Comical

@JürgenA.Erhard I don't think anyone doubts that unit tests can detect a superset of the errors a type system can. Type systems shine when refactoring a codebase, you get immediate feedback on what your changes will break whereas your unit tests will take for ever to run and might not detect everything if you have incomplete code coverage. Then you not only have to fix the code, but also the tests. In my experience off by one and similar trivial bugs happen only once with fresh code. I typically discover them testing manually and usually don't even bother to write regression tests for them. – Ujiji

Answer 3

For any reasonably sized project, you just cannot account for all situations with unit tests only.

So my answer is "no", and even if you manage to account for all situations, you've thereby defeated the whole purpose of using a dynamic language in the first place.

If you want to program type-safe, better use a type-safe language.

Skald

I don't want to program type-safe, I want to program bug-free. – Comical

When it comes to type-safety: you can run, but you can't hide. If you want your code to be free of type-related bugs, either you do the work, or you let the compiler do most of it. Imagine you write myObject.myAttribute, but myObject doesn't even have this attribute. How to ensure it doesn't blow up? Well, either you need to write code that handles the error, or you first check if the attribute exists. Yes, a unit test could find this issue, but if you have to write unit tests to spot for all possible typing errors, you've got a maintenance nightmare if you ask me. – Skald

Answer 4

No.

But that's not the most important question, the most important question is: does it matter that it can't?

Consider the purpose of static type checking: avoiding a class of code defects (bugs). However, this has to be weighed in the context of the larger domain of all code defects. What matters most is not a comparison along a narrow sliver but a comparison across the depth and breadth of code quality, ease of writing correct code, etc. If you can come up with a development style / process which enables your team to produce higher quality code more efficiently without static type checking, then it's worth it. This is true even in the case where you have holes in your testing that static type checking would catch.

Olin

Answer 5

Having 100% code coverage doesn't mean you have fully tested your application. Consider the following code:

if (qty > 3)
{
    applyShippingDiscount();
}
else
{
    chargeFullAmountForShipping();
}

I can get 100% code coverage if I pump in values of qty = 1 and qty = 4.

Now imagine my business condition was that "...for orders of 3 or more items I am to apply a discount to the shipping costs..". Then I would need to be writing tests that worked on the boundaries. So I would design tests where qty was 2,3 and 4. I still have 100% coverage but more importantly I found a bug in my logic.

And that is the problem that I have with focusing on code coverage alone. I think that at best you end up with a situation where the developer creates some initial tests based on the business rules. Then in order to drive up the coverage number they reference their code when design new test cases.

Anthrax

Sorry, I should clarify. I'm probably misusing the term, but by "coverage", I meant something more along the lines of "checking all possible states" instead of "lines of code executed". Either way, this wouldn't be caught by type-checking either, so it's somewhat irrelevant to the current question. – Manor

Answer 6

Given all the benefits of dynamic, late-binding languages, I suppose that's one of the values offered by Unit Tests. You'll still need to code carefully and intentionally, but that's the #1 requirement for any kind of coding IMHO. Being able to write clear and simple tests helps prove the clarity and simplicity of your design and your implementation. It also provides useful clues for those who see your code later. But I don't think I'd count on it to detect mismatched types. But in practice I don't find that type-checking really catches many real errors anyway. It's just not a type of error I find occurring in real code, if you have a clear and simple coding style in the first place.

For javascript, I would expect that jsLint will find almost all type-checking issues. primarily by suggesting alternate coding styles to decrease your exposure.

Correct

Answer 7

Manifest typing (which I suppose you mean) is a form of specification, unit testing is much weaker since it only provides examples. The important difference is that a specification declares what has to hold in any case, while a test only covers examples. You can never be sure that your tests cover all boundary conditions.

People also tend to forget the value of declared types as documentation. For example if a Java method returns a List<String>, then I instantly know what I get, no need to read documentation, test cases or even the method code itself. Similarly for parameters: if the type is declared then I know what the method expects.

The value of declaring the type of local variables is much lower since in well-written code the scope of the variable's existence should be small. You can still use static typing, though: instead of declaring the type you let the compiler infer it. Languages like Scala or even C# allow you to do just this.

Some styles of testing get closer to a specification, e.g. QuickCheck or it's Scala variant ScalaCheck generate tests based on specifications, trying to guess the important boundaries.

Benzol

List<String> tells you next to nothing about the semantics of this list. What do these strings mean? A unit test (an "example"… or rather, many) does. And the docstring/documentation will too. – Comical

Answer 8

I suppose it could if you are very thorough. But why bother? If the language is already checking to ensure static types are correct, there would be no point in testing them (since you get it for free).

Also, if you are using static typed languages with an IDE, the IDE can provide you with errors and warnings, even before compiling to test. I am not certain there are any automated unit testing applications that can do the same.

Impromptu

Answer 9

I would word it a different way--if you don't have a statically-typed language, you had better have very thorough unit tests if you plan on doing anything "real" with that code.

That said, static typing (or rather, explicit typing) has some significant benefits over unit tests that make me prefer it generally. It creates much more understandable APIs and allows for quick viewing of the "skeleton" of an application (i.e. the entry points to each module or section of code) in a way that is much more difficult with a dynamically-typed language.

To sum up: in my opinion, given solid, thorough unit tests, the choice between a dynamically-typed language and a statically-typed language is mostly one of taste. Some people prefer one; others prefer the other. Use the right tool for the job. But this doesn't mean they're identical--statically-typed languages will always have an edge in certain ways, and dynamically-typed languages will always have an edge in certain different ways. Unit tests go a long way towards minimizing the disadvantages of dynamically-typed languages, but they do not eliminate them completely.

Runaway

If you have your typical statically-typed language, you still need tests (which, most of the time these days, will mean unit tests). – Comical