Jenkins failed unit CanExecute test's methods nondeterministic

Question

We have a lot CanExecute tests for a various commands in our project. All tests passed properly when we use Visual Studio testing or AxoCover.

We tried to add some previous object initialization, before executing 'CanExecute' and sometimes it worked (or we thought that).

testedViewModel.Object.InEditMode = inEditMode;

I have a test:

[TestCase(true, true, TestName = "Command_InEditMode_CanExecute")]
[TestCase(false, false, TestName = "Command_NotInEditMode_CannotExecute")]
public void CommandCanExecute(bool inEditMode, bool expectedResult)
{
    var testedViewModel =
        new Mock<SomeViewModel>(inEditMode)
        {
            CallBase = true
        };

    testedViewModel.Setup(x => x.InEditMode).Returns(inEditMode);

    Assert.AreEqual(expectedResult, testedViewModel.Object.Command.CanExecute(null));
}

Sometimes (NOT ALWAYS) when Jenkins does the build and run unit tests some of can execute tests failed with message:

MESSAGE:
  Expected: True
  But was:  False

+++++++++++++++++++  
STACK TRACE:
   at Project.CommandCanExecute(Boolean inEditMode, Boolean expectedResult)

The problem is that is happening only on Jenkins and it's very nondeterministic.

EDIT:

Ok, one more thing to think about. Property InEditMode is placed in base parent class of SomeModelView.

And I merged code for you in the sample.

public BaseViewModel 
{
    public virtual bool InEditMode {get; set;}
}

public SomeViewModel : BaseViewModel
{
    public SomeViewModel () : base ()
    {

    }

    public ICommand Command { get; set; }

    public virtual void RegisterCommands()
    {
        Command = new RelayCommand(/*Do something*/, () => InEditMode);
    }
}

And we think that can be related, that object is thinking that is initialized before initialization of base class is done. But that is very hard to check this with a Jenkins.

---

SOLUTION

I've created an attribute class:

[AttributeUsage(AttributeTargets.Method | AttributeTargets.Class | AttributeTargets.Interface | AttributeTargets.Assembly, AllowMultiple = true)]
public class GarbageCollectorDisabler : Attribute, ITestAction
{
    public void BeforeTest(ITest test)
    {
        GC.TryStartNoGCRegion(2048 * 4096);
    }

    public void AfterTest(ITest test)
    {
        GC.EndNoGCRegion();
    }

    public ActionTargets Targets => ActionTargets.Test;
}

And then I can use for each 'CanExecute' test this attribute:

[GarbageCollectorDisabler]
[TestCase(TestName = "SomeTest_InEditMode_CanExecute")]
public void SomeTestCanExecute()
{}

Krupp

Do you have racing conditions in your code? Sometimes we have a similar because our tests are executed in multiple threads – Stableboy

No, the code you see above it's almost completely the same in our project. We have very simple CanExecute sometimes, like above. And our Jenkins does not executed tests in multiple threads as far as I know. – Zeringue

@PiotrZieliński The messaged says : MESSAGE: Expected: True But was: False the test you have showed inputs false every time for the expected result, so how could the expected be true in the message you showed us? – Glovsky

also please show the CanExecute method on the command – Glovsky

I added second test case. CanExecute method is in the code. It's only that: () => InEditMode – Zeringue

@PiotrZieliński How is it that InEditMode is able to be overridden in your example when it is not virtual? Provide a minimal reproducible example that can be used to reproduce the problem, allowing a better understanding of what is being asked. – Infinitive

AFAIK you need a virtual or an interface or when you moq one - it'll error otherwise. Do you really have a property rather than method for canexecute? Because you could just set that. And.. What's that Object.Command all about? – Tingaling

testedViewModel.Object is an instance of mocked object. And Command is a name of property in the class which contains RelayCommand object with an anonymous function. And you are right, InEditMode is a virtual type, forgot to add this in code above. – Zeringue

Have you tried rewriting to not use parameters in method tests at all? – Gosling

How it can help here? And then had I write all test cases in different tests? It's not something I looking for I'm afraid. – Zeringue

Can you try running with a single test to see if there's any issue ? If it runs every time with a single test, we can rule out the multi-threading / test-interleaving flows – Byler

Answer 1

Smells like a garbage collection issue to me. I don't see anything that jumps out at me in your sample, though the code sample is incomplete (where is RegisterCommands invoked?) so something crucial could be missing.

See the source for RelayCommand.CanExecute(). It takes a weak reference to the action you pass in, and once that action is collected CanExecute will return false. See my answer here for an example of this happening.

I reiterate @Nkosi's comment, create a minimal example rather than showing us bits and pieces.

Nagy

Yes, it was a garbage collector issue! I will add solution to my original post. – Zeringue

@PiotrZieliński Not a huge fan of that solution. Your code has a bug (some object is not being retained correctly) and at some point you'll hit it in a non-test environment. Disabling the GC during test will just mask the bug. – Nagy