Assert.Inconclusive and IgnoreAttribute

Question

What is the right way to use Assert.Inconclusive and IgnoreAttribute in MS Unit test framework?

We are using Assert.Inconclusive mainly for tests which are:

• Not implemented yet
• Somehow broken or incomplete = requires futher attention
• When test body is for any reason commented out

We are doing this because:

• Inconclusive test can have message
• We want to see such tests in test results on TFS

Our problem is that Inconclusive tests are marked as error in both TFS and Resharper. If we use IgnoreAttribute instead we will see these tests in Resharper but MS Test runner and TFS will ignore them at all. Using IgnoreAttribute in TFS and MS Test runner is same like commenting whole test which is useless.

Neolatin

Answer 1

I like to think how you are describing Inconclusive is the proper usage.

Though in my experience, Inconclusive is treated more like a warning than an error. In fact, they are reported in the TRX separately from errors:

<TestRun>
   <ResultSummary outcome="Inconclusive">
      <Counters total="1" 
          executed="0" error="0" failed="0" 
          timeout="0" aborted="0" inconclusive="1" 
          passedButRunAborted="0" notRunnable="0" 
          notExecuted="0" disconnected="0" 
          warning="0" passed="0" completed="0" 
          inProgress="0" pending="0" />

I typically run the mstest executable from an <Exec> task in my msbuild script and then peek inside the TRX to determine whether it should fail the build.

Debrahdebrecen

Yes, this is the best workaround I've read about this problem. I created a TFS template so that the thought described here is integrated into TFS build process. blog.dbtracer.org/2011/02/27/… – Emmi

@PetrKozelek, unfortunately,your blog is not accessible – Youngster

@PetrKozelek I found this response, as well as a post you left on Ewald Hoffman's blog that referenced "Inconclusive Tests in TFS Build Should Not Break the Build." Is your blog hosted anywhere now? I'd like to see what you did. – Sexagenarian

Shame about Petr's blog. Mine is accessible, though the closest I've come to writing about this is here: bryancook.net/2011/05/tests-are-broken-now-what.html At the time I answered this, I was using TFS10 and using the UpgradeTemplate.xaml build template. It's crude in comparison to the newer templates, but it's effectively an MSBuild script. The XmlPeek task can be used to inspect the TRX and the Error task can fail the build if the error or failed count is not zero. – Debrahdebrecen

Answer 2

i also see a dilemma in the current implementation.

• Inconclusive assertions are included in the TRX report, but mstest.exe (and also vstest.console.exe) will return 1 (meaning error) after execution.
• TestMethods with the Ignore attribute will not be reported as an error, but they are completely hidden from the TRX report.

my personal understanding is as follows:

use the [Ignore] attribute to (temporarily) disable / skip the method:

[TestMethod]
[Ignore] // <== disabled through "Ignore" attribute
public void Test001()
{
   //execute some stuff ...
   Assert.IsTrue(...);

   //execute some stuff ...
   Assert.AreEqual(...);
}

do not misuse the Inconclusive assertion for this purpose:

[TestMethod]
public void Test002()
{
    Assert.Inconclusive(); // <== misuse of "Inconclusive" to skip this test

    //execute some stuff ...
}

instead, Inconclusive should be used conditionally: only if we can't tell whether the component to be tested does work as expected or not.
for example in case an external resource we depend on is not available at time of test execution:

[TestMethod]
public void Test003()
{
    //check if the server is running,
    //otherwise can can't test our local client component!
    if (!WebServiceAvailable())
    {
        Assert.Inconclusive(); // <== skip remaining code because the resource is not available
    }

    //execute some stuff ...
    Assert.AreEqual(...);

    //execute some stuff ...
    Assert.AreEqual(...);
}

_ _

conclusion:
to disable / skip a test the logical way is to use the [Ignore] attribute.
i clearly see the current behaviour of mstest.exe not reporting on any ignored test as a bug that should be fixed.

feel free to up-vote the following bug reports:

• https://connect.microsoft.com/VisualStudio/feedback/details/779743/
• http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/2088845

Unstop

Answer 3

As of the MSDN docs:

1. IgnoreAttribute (since VS 2005) means "this test should not be run" see https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.ignoreattribute(v=vs.80).aspx (Archive version on WaybackMachine).

2. Assert.Inconclusive (since VS 2005) means "Indicates that an assertion cannot be proven true or false. Also used to indicate an assertion that has not yet been implemented." see https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.assert.inconclusive(v=vs.80).aspx (Archive on WaybackMachine)

I think these are realy clear statements when you have to use which one.

Tropho

Answer 4

I've been doing some research into this, as well as trying it out at home. The end result is that I believe the [Ignore] attribute for MSTest does indeed leave out the test method completely. I tried looking at settings in Visual Studio to see if there was an override, but could not find anything.

Shame about that, as not seeing the ignored tests is bad since you may end up thinking you have a suite of 100 MSTest tests running nicely, but it turns out that there are 50 which are missing from the results that you never knew about due to the [Ignore] attribute! Urgh.

Photocathode

Answer 5

Classically, in when testing, unit tests should be very specific so that there is a (close to) one-to-one correspondence between features and tests for these features.

To test certain features, the system under test (SUT) first needs to be brought into a certain state. Once that state is reached, the test can actually test the feature that it is meant for. Now, what should be the status of such a test, if already the setup part fails. It cannot make a statement on the functioning of the feature under test, as the test never reached the status requires to exercise the feature.

In such cases, it is common to assign an inconclusive result, as we simply cannot know if the feature works as it should, and so we cannot pass or fail. The fact that the setup itself does not work as expected should be covered by a separate test.

So imagine, I want to test that a 'foo' that has been 'bar'ed will return a 0 when 'qux'ed. This test should not test if 'foo' can be 'bar'ed, so any failure to be 'bar'ed will return an inconclusive, whereas a failure to respond to 'qux' will be proper fail.

Taal

Answer 6

I was having the same issue with VS marking inconclusive tests as failed. I found there's a setting in .runsettings you can set to not mark them as failed. Just set the setting MapInconclusiveToFailed to False. Here's the link of the sample usage.

https://learn.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?branch=release-16.4&view=vs-2019#mstest-element

Vivianaviviane