What are the real-world pros and cons of each of the major mocking frameworks? [closed]

Question

see also "What should I consider when choosing a mocking framework for .Net"

I'm trying to decide on a mocking framework to use on a .NET project I've recently embarked on. I'd like to speed my research on the different frameworks. I've recently read this blog post http://codevanced.net/post/Mocking-frameworks-comparison.aspx and wondered if any of the StackOverflow audience has anything to add in the way of real-world advantages and caveats to the frameworks.

Could people could list the pros/cons of the mocking frameworks they either currently use or have investigated for their own use on .NET projects. I think this would be not only a help to me to decide for my current project, but it will help others make more informed decisions when picking the correct framework for their situation. I'm not an expert on any of the frameworks but I would like to get arguments for and against the major frameworks I've come across:

• RhinoMocks
• Moq
• TypeMock Isolator
• NMock
• Moles

And other usable alternatives that I've missed. I'd also like insights from users that have switched or stopped using products because of issues.

Cothran

Coding software is hard enough. I'd hate to use one that constantly mocks me in the process. – Homosporous

You may find this question and its answers useful: #643120. It also points to several other related questions, particularly #37859. – Ignitron

Thanks Jeff - they're very helpful; I didn't come across them while I was searching for that info. – Cothran

The book artofunittesting.com has lots of good info on this, although the author does have a (clearly noted) bias. – Depside

Answer 1

I don't know Moles at all, but I'll cover the ones I know a bit about (I really need a table for this, though).

Moq

Pros

• Type-safe
• Consistent interface
• Encourages good design

Cons

• Not as full-featured as some of its competitors
  • It can't mock delegates
  • It can't do ordered expectations
  • probably other things I can't think of right now...
• Can only mock interfaces and virtual/abstract members

Rhino Mocks

Pros

• Type-safe
• Full feature set
• Encourages good design

Cons

• Confusing API. There are too many different ways to do the same things, and if you combine them in the wrong way it just doesn't work.
• Can only mock interfaces and virtual/abstract members

TypeMock Isolator

Pros

• Type-safe (AFAIR)
• Can mock anything

Cons

• Very invasive
• Potential Vendor Lock-In
• Does not encourage good design

NMock

Pros

• Encourages good design
• Works on any version of .NET (even 1.1)

Cons

• Not type-safe
• Can only mock interfaces and virtual/abstract members

Please note that particularly the advantages and disadvantages regarding TypeMock are highly controversial. I published my own take on the matter on my blog.

I started out with NMock when that was the only option back in 2003, then migrated to Rhino Mocks because of its type safety, and now use Moq because of the simpler API.

Dipteral

As to the ordered expectations in Moq: Phil Haack shows a nice way here to circumvent that: haacked.com/archive/0001/01/01/moq-sequences.aspx – Distort

@Mark: Can you elaborate on your observation that TypeMock is very invasive? In what way? – Cothran

@Mark: P.S. I read your blog post regarding TypeMock and given the arguments raised by Roy Osherove and your responses [and not knowing anything else about TypeMock], I would have to conclude that your argument seems to be more compelling. – Cothran

@BenAlabaster: It uses the Profiling API to intercept method calls etc. at run-time. Basically, the code that's running is not the code you (or someone else) wrote. – Dipteral

+1 - great answer and great blog post! I'm not clear on what you mean by Moq can't mock delegates, though. It can mock events, and you can always pass a lambda into any method with a delegate parameter (no need for a framework there). – Depside

@TrueWill: Let's say that you have a class that is injected with a delegate instead of a real interface/base class. With Rhino Mocks, you can actually set up expecectations on that and pass a dynamically generated instance. Last time I checked, that wasn't possible in Moq. – Dipteral

@Mark I can think of a bunch of places I pass delegates... so that might make Moq a tough pill to swallow - at least for this project. – Cothran

@BenAlabaster: Well, in practice it turns out to be not so much of an issue, since you can always just pass a lambda expression instead. – Dipteral

@Mark - That's the real world info I was after; Can't mock delegates but can be imitated by passing lambda expressions instead. That's really useful. – Cothran

I have to disagree with the "Encourages good design / Does not encourage good design" evaluations. Besides being higly subjective and controversial, this ignores several obvious real-world situations where the use of limited mocking tools can lead to poor code. For example, a tool that is unable to mock a non-virtual method forces either a separated interface to be created (which most likely will be pointless), or methods not designed for subclassing to be declared as virtual; this is not exactly encouraging good OO design... – Filament

@Rogerio: Well, I disagree, although I know it controversial. It often turns out that the separated interface or virtual method you originally thought only made sense for Testability reasons suddenly become handy in the production code. However, it takes skill to design correctly, and not just hodge-podge to support unit tests. Done correctly, though, Testability is just a special case of the Open/Closed Principle: blog.ploeh.dk/2009/06/05/… – Dipteral

Actually, you are agreeing with me 8^), since you admit that sometimes (you said "It often turns out ...") introducing a separated interface or virtual method may NOT "become handy in the production code". In such cases, therefore, you would want to mock the class without having to do those things. And this is technically possible (easy even) to do both in .NET and in Java. – Filament

Answer 2

So far I have used RhinoMocks and Moq. Moq is currently my favourite due to its simplicity which is currently all I need. RhinoMocks is pretty powerful but I have never been in the position to fully tap into it.

Distort

I started out with RhinoMocks and quickly moved to Moq once I heard about it as well. Moq is just so gosh darn easy to use. – Ploss

Answer 3

We've used Rhino Mocks for more than a year now. PRO:

• easy to create the mocks
• can mock public and internal methods
• can mock interfaces or classes
• can create partial mocks (mocking only specific methods from a class)

AGAINST:

• methods must be at least internal and virtual (can mess with your architecture)
• difficult to use for property-by-property asserts, especially for collections of objects that get created inside the test scope - the constraints syntax gets complicated
• you have to be careful when the recording stops and the playback begins
• careful about what calls are being mocked (like a property call that you didn't see or a method that wasn't virtual) - the errors you may get are not very helpful

As a general note, we've found that using the mocking frameworks promotes "white box" testing (especially for unit tests). We ended up with tests that validated HOW things were done, not WHAT they were doing. They were useless for refactorings and we had to rewrite most of them.

Doloritas

Personally I prefer doing state testing with stubs to behavior testing with verifiable mocks. This can help avoid fragile tests that validate the "how". – Depside

Ooh, having to design your code to accommodate the mocking frame work is a definite deal breaker for me. +1 for your insights on that, very valuable. – Cothran

@Ben - you're going to have that restriction with anything except TypeMock. If you're mocking interfaces there's no problem. With concrete classes, you can only mock out virtual methods. – Depside

@Depside - Good design principles are there because they make your code simpler, more extensible, easier to maintain etc, not to cater to your mocking framework. Following good design principles might make unit testing easier, but catering to your mocking framework in your design decisions risks compromising good design principles and tightly coupling your code and mocking framework. Isn't dependency something we've been trying to get away from since the dawn of Dependency Injection, IoC and factory patterns? Why is it suddenly okay to start designing our code around this dependency? – Cothran

@Ben - If your code follows the Dependency Inversion Principle, you will not encounter this restriction. It's only when you have concrete dependencies that you can't easily decouple (read: legacy code) that you need TypeMock. You don't need (or want) to deploy your mocking framework; it's only used by your unit tests. Your production code should not be coupled to the mocking framework. – Depside

@Depside - I agree, although coupling of production code to the mocking framework for the purpose of deployment wasn't what I meant. What I meant was that if your code is designed around being able to mock your objects with a particular framework rather than using best design practices from a coding standpoint then there is a possibility that it will become harder to switch out your mocking framework for another one if you want to change your testing platform. – Cothran

@Depside - You are right, the Dependency Inversion is what really works. It allows you to stub the real external dependencies of the system like web services and databases and run tests over whole components or the whole system, not just at class level. We've developed a simple in memory "list-based" database that can replace the real database for the tests via dependency injection without affecting the production code. Unit tests at class / method level get in the way of refactorings that extend over several classes. – Doloritas

@Ben - absolutely. The choice of a mocking/isolation framework should not affect the design of production code. – Depside

Answer 4

Like Frank and Chris, I tried RhinoMocks and switched to Moq. I haven't been disappointed. See my series of blog posts:

• Stubbing problems with Rhino Mocks
• Mocks: The Next Generation
• Mocks: The Next Generation II
• Switching to Moq

EDIT: Note that I generally do state-based testing with stubs; I seldom do behavior testing with verifiable mocks.

Depside

Can you elaborate on what pushed you to switch? Was it the stubbing issues you ran into with Rhino Mocks that caused you to jump ship or was it something that Moq was just doing better? – Cothran

@Ben - It was a combination of the stubbing issues and the complexity. With Rhino Mocks I felt like I was fighting against the library (and I was always referencing the documentation); Moq seemed much more natural to me. I'd suggest trying both and sticking with the one that most closely matches your idioms. – Depside

Thanks for referencing your blog posts. I followed along your path and dug through some of the discussions and decided to start with Moq to see how I fare. – Hyderabad

@Hyderabad - thanks! I'm using NSubstitute more recently, and go back and forth on the state vs. behavior testing based on my needs. – Depside

Answer 5

I've not used all those frameworks, but I looked at RhinoMocks and Moq, and went with Moq because it feels more elegant and much simpler. I am using the trunk version which includes a must-have fix for the 4 argument limit imposed on callbacks in the most recent 4.0 beta release.

I especially like the default Moq behavior, which doesn't behave like a strict Mock Object failing tests when unexpected calls are made. You can configure it to do this if you want, but I find that requires me spending way too much time setting up expectations and not enough time testing.

Imprint

Answer 6

I use TypeMock since I'm developing on SharePoint. As TypeMock can mock anything, it's proved a valuable resource when unit testing our SharePoint webparts, event recievers, workflows, etc.

On the downside, TypeMock can be expensive, however there is a version available which is specific for SharePoint and costs less then the full TypeMock package. I highly recommend it.

The one thing I do disagree with is this notion that TypeMock does not make you design your code very well. Often the classes I create, and overall code, are designed well. Just because I use TypeMock doesn't mean I sacrifice the quality of my design - I still practise IoC and SRP. Just because TypeMock can mock anything does't mean I write my code to reflect that ability.

Bosk

Answer 7

You may want to keep in mind that if you need to support a multi-language environment (e.g. VB) all of the code configurable frameworks (I can speak to Moq and RhinoMocks directly) are going to be painful given the (lack of) anonymous delegate/lambda syntax in VB. This will be more possible in Visual Studio 2010/VB 10 but will still not be comparable to the nice C# lambda syntax.

TypeMock appears to have some support for VB

Pigweed

I don't necessarily need cross language compatibility, although that would definitely be a nice bonus feature. That might sway me if two frameworks were otherwise very similar, but it's doubtful that I'd trade that for another useful feature. – Cothran