Does JMockit have any drawbacks at all?

Question

This comparison shows, that JMockit has several advantages over other frameworks.

Are there also any advantages that one of the others (JMock, EasyMock, Mockito, Unitils, PowerMock + Mockito/EasyMock) has over JMockit?

Tubuliflorous

Answer 1

Three drawbacks:

• You must use a Java agent to do bytecode instrumentation.
• You can't use the signed junit.jar file shipped with Eclipse.
• You have to learn a mock API. (In contrast to a stub object)

You can always discuss if it's a good thing to be able to mock a final class like JMockit can. Unless it's legacy code, refactoring is usually a better alternative.

With IDEs like Eclipse, I find myself using tool support to generate stubs inside the test class more frequently than mocking (JMockit, Mockito, etc.) in the recent time. The advantage with this approach is that it's very simple. This is especially nice when you have a team with many developers and some of them don't like testing and have little motivation to learn a mocking framework. Also, stub implementations don't have framework limitations!

If you're open for stubbing as an alternative, you should check out Robert C. Martin's blog about mocking and stubbing here and here

Else, it looks very good! Although I have only experience with JMock, EasyMock and basic knowledge with JMockit.

Durango

Yeah, I'm also asking myself, how far I want to go with mocking. My own code uses Dependency Injection throughout, so mocking isn't really a complex task (though I don't always use interfaces for everything due to the added maintenance effort + performance overhead when used with GWTRPC). When you say you use tool support to generate stubs in Eclipse - which tools do you have in mind? – Tubuliflorous

Just the core Java support in Eclipse. For example if you create a private or anonymous class that implements the interface you want to stub, then Eclipse can create a skeleton implementation. Just click <Ctrl> + 1 after you have selected the class. – Durango

Ah, ok - I thought maybe there's some tool that keeps the stub in sync, when the code changes or something. – Tubuliflorous

1) What exactly you mean by "must use a Java agent"? (The "-javaagent" parameter is optional with JDK 1.6.) 2) Correct about the signed JUnit jar, but it's just as easy to use the "real" JUnit as a "User library". 3) Of course you have to learn a mocking API! The same is true for all other mocking/stubbing APIs in that comparison matrix... what's your point? – Antigone

@Rogerio: Did you read the title of the question? It's not about if JMockit is a good framework or not. It is if it has any drawbacks at all. I personally like mocking and JMockit looks at least from the specification to be the best alternative. 3) I compare a mock API against stub objects (not stub API). And added some links you obviously didn't bother to read before you gave me -1. In some situations, a simple stub object is sufficient. My minor complaints on this questions just says the framework is really good. So keep up the good work on the project! – Durango

@Espen: Yes, I read not only the title but the whole question. And I read those two articles months ago, but they are hardly relevant here. Please note it's not about "any drawbacks at all", but whether JMockit has drawbacks that the other tools in the comparison matrix don't, and which should be added to the matrix. At least, that's what I understand the intent behind the question was. Your answer, however, doesn't seem to address that, as it talks about not using a mocking API at all. BTW, creating stubs by hand (or through an IDE) is full of limitations - not a viable alternative. – Antigone

@Espen: note I have written two responses to the first article referenced, back in January... – Antigone

@Rogerio: Is a stub created by hand full of limitations? I can not see any limitations except those given by Java itself. And since it is plain Java, it's nothing new to learn! Comparing mock frameworks, I will still say using a javaagent with Java 5 is a minus. Just a little one though.. And since JMockit is a very powerful framework, it also forgives bad design. Which is both good and bad. As I have tried to say the whole time, JMockit looks like a great framework. My major point was that plain stub objects in many situations removes the dependencies without learning a new mock framework. – Durango

@Rogerio: JMockit is fantastic, and it's very much possible, that I will use it someday. Will there be an abbreviated version of the JMockit documentation? (hint: All that's important about Mockito can be learnt in less than two hours...) Maybe together with defining a subset of JMockit features, that are marked "safe for good design" (this includes mocking final methods BTW!) That would help, because if we can test everything, it becomes a lot harder to convince team members (or - I'm afraid - even myself) to write good, reusable code without hidden or unnecessary dependencies! – Tubuliflorous

@Espen: Yes, a hand-coded stub or mock is limited, when compared to an equivalent object created by a good mocking library. And such limitations have nothing to do with Java, either the language or the JVM. For example, it forces the tested class to provide some way for the test to pass the mock in: you shouldn't have to add complexity to the tested code in order to write unit tests. Another limitation is that you cannot write a stub for a final POJO class (one which does not implement a separate interface). Neither can you stub/mock constructors or static methods. – Antigone

JMockit does not "forgive" bad design, anymore than any other Java library. You can't prevent bad code by imposing arbitrary technical limitations on developers, nor is the place of reusable APIs to do that. Such tools and APIs are about enabling developers to produce better software, not about constraining them to follow some supposed "right path" dictated by others. To me, and I have written about this elsewhere, the freedom to fully use the Java language, including final classes/methods, constructors, the new operator, static methods, etc. is non-negotiable. – Antigone

There is already an "abbreviated" version of the JMockit documentation, the "Getting Started" page. It's quick to read and should give a new user enough information to start writing tests. Also, the API javadocs contain links to relevant sections in the Tutorial and to sample tests, so it should be easy to learn more as you code from inside a Java IDE. – Antigone

Comments from others above imply that a public Java class which 1) does not implement a separate Java interface, and 2) is marked final would somehow constitute "bad design". This is non-sense, with no supporting evidence in programming literature (that I can find, anyway). And no, the GoF book does not say that a class must implement a separate interface, if you're thinking of the "program to an interface, not an implementation" principle (one that I always follow, BTW). Marking a class final is considered good design if such a class is not designed for inheritance (see Effective Java). – Antigone

@Rogerio: For those situations you're mention here and many others, a mock framework is often the best choice. But sometimes it can also be better to refactor the code and the need for a powerful framework like JMockit isn't longer necessary. All I say is that it is several solutions and strategies. Complex code should be written in methods that implements an interface. That's my personal opinion at least. – Durango

@Rogerio: Remember that you as a author of JMockit can not be totally objective. And you as an expert on mocking will have better results with mocking than a developer relatively new to Java. Anyway, it's my last comment on this post.. I will follow JMockit further and most possibly use it on my next project. My impression so far of JMockit is very good! Absolutely nothing of what Crhis and I have written is about bad functionality or API in JMockit. It is about mocking vs stubbing and strategies to achieve good design on your code with many developers on different skills levels. – Durango

Answer 2

I've recently adopted a project that uses JMockit and I think the quality of the code has certainly suffered as a result of the library's ability to mock out static and private methods.

The tests are very brittle because implementation details contained in private methods are being tested (so if I change how the class does something it can break tests, even if what the class does has not been affected).

The code is also littered with calls to static methods - if the developers had not had the ability to mock these out then I think they would have made more effort to de-couple things a little better.

Illuminator

I don't understand this comment. You say the tests had become brittle after using JMockit b/c the implementation details in private methods were tested. But surely the ability to mock such private methods should have helped you mitigate the problem by actually mocking those private methods rather than aggravate it by testing the private methods? So hasn't JMockit worked out in favor of you by making your tests better than without it? Just curious. – Wanderoo

I think the tests were brittle because if someone refactors a class (eg renames/breaks-up/combines some internal methods, or even completely changes how the class works internally without changing its public API) then the tests that poke around inside the class testing its private methods would break. I don't think a unit test should care how a class does its job, it should only be concerned with whether the job is done correctly or not. Does that make it clearer? – Illuminator

Ah, yes it does. Felt like this comment shares the same sentiment as yours. Cheers! – Wanderoo

Thanks for detailed explanation of pitfalls of JMockit. I think if mockery of static and private methods is banned by one's organization rules it should be ok to use JMockit. As long as their comparison matrix is true (wink wink) – Stegodon

@BhabaniSankarMishra No, tests should not fail just because code changes, they should fail if the code stops behaving correctly. – Illuminator

I never said code change, change is usually meant from a business perspective. The whole point of writing test is to make changes safe, if the tests don't fail what's the point of writing tests. – Crawley

@BhabaniSankarMishra yes I agree with you, if we change what the code does, then the tests should fail, but my answer was specifically about changes that don't affect what the code does, only how it does it. I don't want my tests to fail every time I refactor something, which is what happens if you start testing private methods rather than testing the public API. – Illuminator

Testing private methods is questionable. I use it mostly for old legacy code or badly written new code, when refactoring. But one great thing about JMockit is that you CAN test them, if you want/need. If you use for example Mockito you are screwed. – Balanced