Users of Gallio, what Advantages and Disadvantages have you experienced using this Tool?

Question

For those of you familiar with Gallio, what Advantages and Disadvantages have you experienced using this tool, a so-called Test Automation Platform?

Nicolis

Answer 1

We're using Gallio/MbUnit for a year now. We're quite happy with it, the Gallio guys keep introducing cool new features and the development is active. If you decide to use it, here are some hints/notes:

1. Buy yourself a TestDriven.NET license - I think it is a must for Gallio unit tests, since Resharper test runner doesn't know how to run certain tests + Gallio has one of its test runners targeted for TD.NET.
2. We use Gallio.Echo command line runner for CI scripts/builds. Gallio Icarus - the GUI runner (at least the current version) is pretty unstable and not very user friendly.
3. If you want to use the latest Gallio features (like parallel tests etc), be prepared to have to use the latest daily builds of Gallio. If this is an issue, keep to "official" builds, but we haven't had any real problems with daily build versions.
4. I recommend adding the basic Gallio .DLLs and .EXEs to your source control. And actually reference these in your projects, not ones from GAC. This way you'll avoid any "works on my machine" problems if someone in your team has a different version of Gallio installed on his/her machine.

Clarissaclarisse

Answer 2

I tried Gallio as well and it runs much slower than either TDD.Net or the native NUnit GUI test runner. It is even slower than the Resharper runner!

I should add that I'm talking about running NUnit tests here.

I don't remember the exact numbers but it was taking around 3 minutes for the same tests, that NUnit finished in 30s (running on one thread, single process,multiple domain).

That in itself already makes it a no go. Add to that the bulky GUI and you know to stay way from it.

---

Some extra information:

• In my solution I have NUnit tests and recently started adding MSpec specifications. I have the NUnit Gui open to automatically rerun my old tests (all new tests are written as MSpecs) after I recompile and the test dlls change.

• I use TDD.Net to run my Mspec specifications.

• This enables me to already continue working once my Specs have finished running while the NUnit Gui runner is still finishing.

Prism

Answer 3

When we evaluated Gallio, we experienced stability issues with large projects. Our smaller projects ran beautifully, however. Great concept...I think it will generate a big buzz once it's a bit more refined.

I might want to add that Resharper support was missing (or broken) for a while, but I've heard it's back.

Sig

Yeap, the latest alpha, which I believe is 3, added Resharper 4 support. – Supersession

Answer 4

The latest release of Gallio (3.0.6) has address a lot of the stability issues mentioned in the these posts. In particular, Icarus is much more stable now and has the ability to attach to the debugger built in, so it can be even faster to use than Resharper which recompiles the code before each test run.

Tympanic

Answer 5

It's terrible unstable, I used it about 3-4 months ago it was terrible unstable and slow.

Now I've just tried it and it crashes when you click "Save", then it never opens again unless you go and clean up the "Local Settings", I assume it's still terrible unstable.

I wish they would stop adding new features and instead fix these rather obvious bugs.

P.S. Project got 1000~ unit tests and it's using nUnit (maybe it's just my nUnit and Gallio don't play well together )

I really want to use it and I've got 3.1 - 313, I couldn't even save a project without getting a crash!

After all bad stuff advantages Here:

• Great support for different flavours, I've used it for nUnit and mbUnit it's really good. It even support RowTest in nUnit very well.

• GUI is pretty cool, clean

• Got great features like setting working directory
• Reporting & Integration

Lilia

Please at least report these issues (code.google.com/p/mb-unit/issues/list) otherwise they'll never be fixed. Even better would be providing a patch! – Shitty

That's neat idea although that takes time. Last time I tried it was horrible it didn't worth the time to report the issue, because it was almost totally broken. But I think this time it's hopeful, so next time I've got enough time to report stuff I'll do it. – Lilia

Answer 6

On the advantages front, integration with everything is quite nice :) Seriously, I'm a huge fan of the experimental integration with Visual Studio Team System (screenshots here). I think that really lowers the bar for those new to unit testing.

The only disadvantage from my perspective so far is the fact that it isn't released yet.

Chas

Answer 7

I have been evaluating Gallio and it is truly a great concept. Now I don't have to tell my developers what unit testing framework they have to use, because it has integration with all the major ones that I know of. They can use which ever they are most comfortable with and I get unit tested code.

The Resharper integration is nice also. I can run all the unit tests with Resharper without having to jump out to a command line to run them.

This should definitely make introducing unit testing into shops a lot easier in my opinion. I agree with @David, the downside is that it isn't released yet.

Supersession