How to programmatically run unit tests with Gallio and MBUnit?

Question

I'm trying to programmatically check my unit tests are passing as part of my deployment process. The application uses MBunit and Gallio for it's unit testing framework.

Here's my code:

var setup = new Gallio.Runtime.RuntimeSetup();
setup.AddPluginDirectory(@"C:\Program Files\Gallio\bin");

using (TextWriter tw = new StreamWriter(logFilename))
{
    var logger = new Gallio.Runtime.Logging.TextLogger(tw);
    RuntimeBootstrap.Initialize(setup, logger);

    TestLauncher launcher = new TestLauncher();                
    launcher.AddFilePattern(dllToRunFilename);
    TestLauncherResult result = launcher.Run();
}

Here's the test which is contained in the DLL I'm loading (I've validated this works with the Icarus test runner):

public class Tests
    {
        [Test]
        public void Pass()
        {            
            Assert.IsTrue(true);
        }

        [Test]
        public void Fail()
        {
            Assert.Fail();
        }
    }

When I run the application I get the following values in results

Which is incorrect as there are indeed tests to run! The log file has the following in it

Disabled plugin 'Gallio.VisualStudio.Shell90': The plugin enable condition was not satisfied. Please note that this is the intended behavior for plugins that must be hosted inside third party applications in order to work. Enable condition: '${process:DEVENV.EXE_V9.0} or ${process:VSTESTHOST.EXE_V9.0} or ${process:MSTEST.EXE_V9.0} or ${framework:NET35}'. Disabled plugin 'Gallio.VisualStudio.Tip90': The plugin depends on another disabled plugin: 'Gallio.VisualStudio.Shell90'.

How do I resolve this issue and find the results to the tests?

Sabbath

What version of Visual Studio? – Culler

I'm working in VS2012 – Sabbath

I have not worked with Gallio before so I am not sure if this would matter, but your Tests class is not a [TestFixture]. Maybe Gallio is not seeing reading for the [Test] becuase it can't find the fixture. – Tymothy

@Tymothy I've removed the TestFixture attribute and it's made no difference. I've also confirmed the DLL with the Icarus test runner - it's behaving as expected. – Sabbath

Answer 1

This works for me, note I used this GallioBundle nuget to get gallio and mbunit, so perhaps there is some difference to what you have installed.

The log messages regarding plugins are expected, those plugins wont work if you are self-hosting the Gallio runtime.

using System;
using System.IO;
using Gallio.Runner;
using Gallio.Runtime;
using Gallio.Runtime.Logging;
using MbUnit.Framework;

public static class Program
{
    public static void Main()
    {
        using (TextWriter tw = new StreamWriter("RunTests.log"))
        {
            var logger = new TextLogger(tw);
            RuntimeBootstrap.Initialize(new RuntimeSetup(), logger);

            TestLauncher launcher = new TestLauncher();
            launcher.AddFilePattern("RunTests.exe");
            TestLauncherResult result = launcher.Run();
            Console.WriteLine(result.ResultSummary);
        }
    }
}

public class Tests
{
    [Test]
    public void Pass()
    {
        Assert.IsTrue(true);
    }

    [Test]
    public void Fail()
    {
        Assert.Fail();
    }
}

Tested like this:

› notepad RunTests.cs
› nuget.exe install -excludeversion GallioBundle
  Installing 'GallioBundle 3.4.14.0'.
  Successfully installed 'GallioBundle 3.4.14.0'.
› cd .\GallioBundle\bin
› csc ..\..\RunTests.cs /r:Gallio.dll /r:MbUnit.dll
  Microsoft (R) Visual C# Compiler version 12.0.21005.1
  for C# 5
  Copyright (C) Microsoft Corporation. All rights reserved.    
› .\RunTests.exe
  2 run, 1 passed, 1 failed, 0 inconclusive, 0 skipped

Pilliwinks

No joy I'm afraid - I've got the Gallio and MBUnit from their website but I'm getting "Could not resolve component for service type 'Gallio.Runner.Projects.ITestProjectManager' because there do not appear to be any components registered and enabled for that service type." – Sabbath

I've actually managed to programmatically run NUnit tests which I'm happy to use so don't panic too much. – Sabbath

I've marked this as the answer because although I can't get it to work it matches other documentation I've seen and the OP has provided a full example – Sabbath

Answer 2

These are instruction for running MBUnit tests in Visual Studio 2012 and above using a neat NUnit trick.

Firstly, install the NUnit Test Adapter extension (yes, NUnit)

• Tools > Extension and Updates > Online > search for NUnit > install NUnit Test Adapter.
• You may need to restart the Visual Studio IDE.

Then, you simply need to add a new NUnit test attribute to your test methods. See example code here (notice the using statements at the top) ...

//C# example
using MbUnit.Framework;
using NuTest = NUnit.Framework.TestAttribute;

namespace MyTests
{
    [TestFixture]
    public class UnitTest1
    {
        [Test, NuTest]
        public void myTest()
        {
            //this will pass
        }
    }
}

You can run and debug the test in visual studio as NUnit and Gallio Icarus GUI Test Runner will run them as MBUnit (enabling parallel runs for example). You will need to stop Gallio from running the NUnit tests by deleting the NUnit folder in the gallio install location i.e. C:\Program Files\Gallio\bin\NUnit

Drouin