Why my C++/CX unit-test does not fail?

Question

I have the following "Windows Store/Metro Style" test methods in VS 2012 C++/CX

This one succeeds, which is ok

TEST_METHOD(TestMethod)
{
    bool passed = false;
    concurrency::event finished;
    finished.reset();                                   
    auto testTask = concurrency::create_task( [&finished, &passed]()
    {   
        passed = true;
        finished.set();
    }); 
    finished.wait(100000);
    Assert::IsTrue(passed);
}

This one fails, which is also ok:

TEST_METHOD(TestMethod)
{
    bool passed = true;
    concurrency::event finished;
    finished.reset();                                   
    auto testTask = concurrency::create_task( [&finished, &passed]()
    {   
        passed = false;
        finished.set();
    }); 
    finished.wait(100000);
    Assert::IsTrue(passed);
}

But for some reason, this test does not fail:

TEST_METHOD(FailedTest)
{
    concurrency::event finished;
    finished.reset();                                   
    auto testTask = concurrency::create_task( [&finished]()
    {   
        Assert::IsTrue(false);
        finished.set();
    });
    finished.wait(100000);          
}

Am I doing something wrong?

As a side note, a possible work-around could be to put all the results of my tests in variables and "test" them all after finished.wait(100000);, but I still would like to know if there is something actually wrong with what I am doing.

Emplace

Should it fail? Should finished.wait call testTask? – Smoko

@MrLister I strongly believe it should fail. Please see my edit. – Emplace

When I run this, the test does fail. The process is terminated unexpectedly, so the failure appears in the Test output window, not in the Test Explorer (the test appears as grayed out in the Test Explorer, I suppose because it did not run to completion). It looks like maybe the test framework only really handles assertions from the main thread? – Galenical

@JamesMcNellis Thanks for replying. That seems to be the case. I just saw the message "The active Test Run was aborted because the execution process exited unexpectedly." in the output window. Initially I thought the test succeeded because it remains in the "Passed Tests" list, I did not notice it was grayed out. On top of that the window that says "Test Passed" from a previous attempt remains there, which was a bit confusing. – Emplace

Answer 1

The test does fail. If you open the Tests Output window (View -> Output, then under Show output from, select Tests), you should see the following:

The active Test Run was aborted because the execution process exited unexpectedly. To investigate further, enable local crash dumps either at the machine level or for process vstest.executionengine.appcontainer.x86.exe. Go to more details: http://go.microsoft.com/fwlink/?linkid=232477

In the Test Explorer, the test should appear grayed out because the test failed to run to completion. Unfortunately, because the test run fails completely, if the test had at any time in the past passed, the test still appears in the Passed Tests group and still has a green check mark next to it. Rely on the grayed-out name as an indication that the test did not run to completion.

I would recommend opening a bug on Microsoft Connect concerning the confusing UI in the Test Explorer in this scenario (if you open a bug, please post the link either as a comment to this answer or in your question, so others can find it).

Galenical

Thanks. I opened a bug here: Test Explorer is not updated properly when a test fails inside an async task – Emplace