Why do my tests fail when run together, but pass individually?

Question

When I write a test in Visual Studio, I check that it works by saving, building and then running the test it in Nunit (right click on the test then run).

The test works yay... so I Move on...

Now I have written another test and it works as I have saved and tested it like above. But, they dont work when they are run together.

Here are my two tests that work when run as individuals but fail when run together:

using System;
using NUnit.Framework;
using OpenQA.Selenium.Support.UI;
using OpenQA.Selenium;

namespace Fixtures.Users.Page1
{
    [TestFixture]
    public class AdminNavigateToPage1 : SeleniumTestBase
    {
        [Test]
        public void AdminNavigateToPage1()
        {
            NavigateTo<LogonPage>().LogonAsCustomerAdministrator();
            NavigateTo<Page1>();
            var headerelement = Driver.FindElement(By.ClassName("header"));

            Assert.That(headerelement.Text, Is.EqualTo("Page Title"));
            Assert.That(Driver.Url, Is.EqualTo("http://localhost/Page Title"));
        }

        [Test]
        public void AdminNavigateToPage1ViaMenu()
        {
            NavigateTo<LogonPage>().LogonAsCustomerAdministrator();
            Driver.FindElement(By.Id("menuitem1")).Click();
            Driver.FindElement(By.Id("submenuitem4")).Click();
            var headerelement = Driver.FindElement(By.ClassName("header"));

            Assert.That(headerelement.Text, Is.EqualTo("Page Title"));
            Assert.That(Driver.Url, Is.EqualTo("http://localhost/Page Title"));
        }
    }
}

When the second test fails because they have been run together

Nunit presents this:

Sse.Bec.Web.Tests.Fixtures.ManageSitesAndUsers.ChangeOfPremises.AdminNavigateToChangeOfPremises.AdminNavigateToPageChangeOfPremisesViaMenu: OpenQA.Selenium.NoSuchElementException : The element could not be found

And this line is highlighted:

var headerelement = Driver.FindElement(By.ClassName("header"));

Does anyone know why my code fails when run together, but passes when run alone?

Any answer would be greatly appreciated!

Programme

Are you sure the tests aren't sharing state? Does the browser close down and re-open between the tests? Which test fails when they're run together? Do BOTH fail, or does one pass, and one fail? – Climber

The browser closes down and so does the command window.. they reopen again to process the following test... Its the first test that pass's then all following tests generally fail and they allways fail for the same reason posted above. – Programme

Doesn't help your specific question, but check out ncrunch.net, it runs your tests in the background. Get rid of the whole "saving, building and then running the test" hassle. – Stephniestepladder

This is happening to me all the time (though with a different error), although mine are hitting a database and running each test in a transaction.. I get non thread safe exceptions so when running through Visual Studio the next test must start before the previous test has finished (running command line does not seem to have the same problem) – Dave

I detailed how to sometimes fix issues with cached module state in Python here: #55067915 – Idealism

Answer 1

Two things you can try

1. put the break point between the following two lines. And see which page are you in when the second line is hit

2. Introduce a slight delay between these two lines via Thread.Sleep

   Driver.FindElement(By.Id("submenuitem4")).Click();
   var headerelement = Driver.FindElement(By.ClassName("header"));
   

Quarterly

tried the thread.sleep function where you suggested. before i added it, when i run the tests it was always the second test that failed, but with the command sleep it was always the second test that past and the first one failed? strange! – Programme

So this might mean that you need the thread.sleep above that "header" element finder, to give enough time for the browser to load the page before your drive look for that element. Try the same in first test too. – Quarterly

ok i added the sleep function to both tests and they have passed! strange i tried it before but never added them to both.. not sure why they pass now but im chuffed nether the less! thanks for your help. – Programme

thread.sleep may not be the answer to 'why this is happening' question. My experience with css based menu clicks with selenium drivers is not that predictable. Thread.Sleep seem to be the only way I was able to reliably process those clicks – Quarterly

Answer 2

Such a situation normally occurs when the unit tests are using shared resources/data in some way.

1. It can also happen if your system under test has static fields/properties which are being leveraged to compute the output on which you are asserting.
2. It can happen if the system under test is being shared (static) dependencies.

Decal

I'm having this situation with InMemory Database in .net core -- Just not sure how to fix it – Asch

To avoid this issue, and to make sure the test is self-contained from a readability perspective, I typically don't rely on shared objects between unit tests. I create separate objects inside each unit test. You can have separate methods for creating separate objects (to avoid rewriting the same code again and again). – Decal

this was it for me, I had a shared public property on the test class that represented some base data starting point. I had to init this object in each test to get rid of the error – Santa

I activated a translation in a pytest unit test and was not aware that it was not set back after the test. A later test failed because it relied on the base language. In Django, solution was to use with translation.override(...) context manager. – Meilen

Answer 3

If none of the answers above worked for you, i solved this issue by adding Thread.Sleep(1) before the assertion in the failing test...

Looks like tests synchronization is missed somewhere... Please note that my tests were not order dependant, that i haven't any static member nor external dependency.

Archaeo

+1 because you made me realise that my static object was the issue and cause of my errors. Thank you! – Elope

Answer 4

look into the TestFixtureSetup, Setup, TestFixtureTearDown and TearDown.
These attributes allow you to setup the testenvironment once, instead of once per test.

Faison

what do you recommend i can clean up in the TestFixtureTearDown? – Programme

@GraemeSecondwave Unfortunately I do not know Selenium. I do however know that when I have run tests together they fail and when I add the common instructions to the setup it works. For instance, what happends if you add NavigateTo<LogonPage>().LogonAsCustomerAdministrator(); to a TestFixtureSetup? – Faison

If the var headerelement... line is highlighted it is probably the line before it that is throwing an exception. Make sure that submenuitem4 exists. Have you tried debugging it? (Right click your test, choose Test With -> Debugger) – Faison

Answer 5

Without knowing how Selenium works, my bet is on Driver which seems to be a static class so the 2 tests are sharing state. One example of shared state is Driver.Url. Because the tests are run in parallel, there is a race condition to set the state of this object.

That said, I do not have a solution for you :)

Christine

Answer 6

Are you sure that after running one of the tests the method

NavigateTo<LogonPage>().LogonAsCustomerAdministrator();

is taking you back to where you should be? It'd seem that the failure is due to improper navigation handler (supposing that the header element is present and found in both tests).

Corymb

well watching the driver navigate through my test it loads up all the pages and navigates to the page as intended. and the page title im looking for displays. it doesnt like the 'Header element' as the test as it is passes when run by it self? – Programme

Answer 7

I think you need to ensure, that you can log on for the second test, this might fail, because you are logged on already?

-> putting the logon in a set up method or (because it seems you are using the same user for both tests) even up to the fixture setup -> the logoff (if needed) might be put in the tear down method

     [SetUp]
     public void LaunchTest()
     {
        NavigateTo<LogonPage>().LogonAsCustomerAdministrator();
     }

     [TearDown]
     public void StopTest()
     {
        // logoff
     }
     [Test]
     public void Test1()
     {...}
     [Test]
     public void Test2()
     {...}

If there are delays in the DOM instead of a thread.sleep I recommend to use webdriver.wait in combination with conditions. The sleep might work in 80% and in others not. The wait polls until a timeout is reached which is more reliable and also readable. Here an example how I usually approach this:

    var webDriverWait = new WebDriverWait(webDriver, ..);
    webDriverWait.Until(d => d.FindElement(By.CssSelector(".."))
        .Displayed))

Bergstein

Answer 8

I realize this is an extremely old question but I just ran into it today and none of the answers addressed my particular case.

Using Selenium with NUnit for front end automation tests.

For my case I was using in my startup [OneTimeSetUp] and [OneTimeTearDown] trying to be more efficient.

This however has the problem of using shared resources, in my case the driver itself and the helper I use to validate/get elements.

Maybe a strange edge case - but took me a few hours to figure it out.

Salazar

Answer 9

I had a similar problem with C++. It turned out I was using uninitialized variable values, which caused undefined behavior, varying upon contents of unreserved memory.

I suspected some kind of memory problem and decided to run the unit test with valgrind.

Valgrind gave errors of the form:

==1795== Conditional jump or move depends on uninitialised value(s)

And indeed, my code was using uninitialized values, which of course causes undeterministic behavior, depending on what garbage happens to be in that memory location when running the code.

The type of bug I had was like this:

    class MyClass
    {
        public:
        MyClass();
        void InitStuff(const bool flagIn)
        
        private:
        bool flag;
    };
    
    void MyClass::InitStuff(const bool flagIn)
    {
        if (flag) // Bug! I'm using member variable flag, not flagIn
        {
            DoStuff();
        }
        flag = flagIn;
    }

So, I suggest checking your code for usage of uninitialized values.

Esperanto

Answer 10

One way in which tests fail when run together, but pass individually is with DateTime comparisons. A thread sleep helps here.

Avra

Answer 11

I fixed it by changing the list MockData to a function. So, each time we use the list data source, it returns a new instance.

My old code:

public static class MockData {
    private static List<PeopleEntity> AllPeople =
    [...]
}

My new code:

public static class MockData
{
    private static List<PeopleEntity> AllPeople =>
    [...]
}

Replete