Qunit test alternates between pass and fail on page refresh

Question

I have a two tests that are causing side effects with each other. I understand why as I am replacing a jQuery built-in function that is being called internally in the second test. However what I don't understand is why the test alternately passes and fails.

This question is similar However, I am not doing anything directly on the qunit-fixture div.

Here are my tests

test('always passing test', function() { // Always passes
    var panelId = '#PanelMyTab';

    var event = {};
    var ui = {
        tab: {
            name: 'MyTab',
        },
        panel: panelId,
    };

    $('<div id="' + panelId + '">')
        .append('<a href="#" class="export">Test</a>')
        .append('<a href="#" class="showForm">Show Form</a>')
        .appendTo('#qunit-fixture');

    jQuery.fn.on = function(event, callback) {
        ok(this.selector == panelId + ' .export', 'Setting export click event');
        equal(callback, tickets.search.getReport, 'Callback being set');
    };

    loadTab(event, ui);
});

test('alternately passing and failing', function() { // Alternates between passing and failing on page refresh
    expect(5);

    var testUrl = 'test';
    $('<div class="ui-tabs-panel">')
        .append('<a href="'+ testUrl + '" id="getReport">Get Report</a>')
        .append('<form action="notest" target="" class="ticketSearch"></form>')
        .appendTo('#qunit-fixture');

    // Setup form mocking
    $('form.ticketSearch').submit(function() {
        var urlPattern = new RegExp(testUrl + '$');
        ok(urlPattern.test($(this).prop('action')), 'Form action set to link href');
        equal($(this).prop('target'), '_blank', 'Open form on a new page');
    });

    var event = {
        target: 'a#getReport',
    };

    var result = getReport(event);

    var form = $('form.ticketSearch');

    ok(/notest$/.test($(form).prop('action')), 'Making sure action is not replaced');
    equal($(form).prop('target'), '', 'Making sure that target is not replaced');

    ok(false === result, 'click event returns false to not refresh page');
});

The tests will start off passing but when I refresh they will alternate between passing and failing.

Why is this happening? Even adding GET parameters to the url result in the same behavior on the page.

In the failing cases, the test is failing because internal jQuery is calling .on() when the submit() handler is set. But why isn't the test always failing in that case? What is the browser doing that a state is being retained during page refresh?

Update:

Here is the code that is being tested:

var tickets = function() {
    var self = {
        loadTab: function(event, ui) {
            $(panel).find('.export').button().on('click', this.getReport);
        },

        search: {
            getReport: function(event) {
                var button = event.target;
                var form = $(button).closest('div.ui-tabs-panel').find('form.ticketSearch').clone(true);

                $(form).prop('action', $(button).prop('href'));
                $(form).prop('target', '_blank');

                $(form).submit();

                return false;
            }
        }
    };

    return self;
}();

Dollfuss

I think it need to be more clear first. Please do lint the code. Also getReport function is clue. Please show getReport function. – Jaquelynjaquenetta

Added the code that is being tested. – Dollfuss

Answer 1

I've modified @Ben's fiddle to include your code with both of your tests. I modified some of your code to make it run correctly. When you hit the run button all of the tests will pass. When you hit the run button again, the second test ("alternately passing and failing") will fail -- this is basically simulating your original issue.

The issue is your first test ("always passing test") alters the global state by replacing the jQuery.fn.on function with an overridden one. Because of this, when the tests are run in order, the second test ("alternately passing and failing") uses the incorrect overridden jQuery.fn.on function and fails. Each unit test should return the global state back to its pre-test state so that other tests can run based on the same assumptions.

The reason why it's alternating between pass and fail is that under the hood QUnit always runs failed tests first (it remembers this somehow via cookie or local storage, I'm not exactly sure). When it runs the failed tests first, the second test runs before the first one; as a result, the second test gets jQuery's native on function and works. When you run it a third time, the tests will run in their "original" order and the second test will use the overridden on function and fail.

Here's the working fiddle. I've add the fix to "un-override" the on function after the test by caching the original var jQueryOn = jQuery.fn.on; function and resetting it at the end of the test via: jQuery.fn.on = jQueryOn;. You can probably better implement this using QUnit's module teardown() method instead.

You can check out https://github.com/jquery/qunit/issues/74 for more info.

Agamic

Answer 2

I'm not sure I can solve this without some more info, but I can point out some possible issues.

The first test seems to have invalid syntax on line 2

var panelId = '#PanelMyTab');

But that's probably a type mistake, seeing as you say the first always passes.

I'm assuming that for the first test to pass(and be valid) the loadTab(event,ui) must run the jQuery.fn.on(), without it no assertions have been run. Which doing some testing with jQuery UI Tabs, seems to be the case (just not sure if it was your intention).

I'm not sure it's advisable putting these assertions within that function, and you must understand that you have overwritten the jquery function with a function that doesn't do anything, so it's likely to cause issues.

You seem to be doing something similar in the second test, you are expecting 5 assertions, but I can only see how the final 3 can be run

ok(/notest$/.test($(form).prop('action')), 'Making sure action is not replaced');
equal($(form).prop('target'), '', 'Making sure that target is not replaced');
ok(false === result, 'click event returns false to not refresh page');

The other 2 are within a submit function that doesn't look like it is invoked as part of the test.

Remember these tests are synchronous so it won't wait for you to hit submit before running the test and failing.

Here is an example

test('asynchronous test', function() {
    setTimeout(function() {
        ok(true);
    }, 100)
})

Would fail as the ok is run 100ms after the test.

test('asynchronous test', function() {
    // Pause the test first
    stop();

    setTimeout(function() {
        ok(true);

        // After the assertion has been called,
        // continue the test
        start();
    }, 100)
})

The stop() tells qunit to wait and the start() to go!

There is also a asyncTest() detailed in the api here

Finally, it seems like you are trying to debug your code with these tests. It would be much easier to use chrome developer tools or firebug in firefox to set breakpoints on your code, and use console.log() and console.dir() to output information.

That being said I have no idea how it works for you at all, so I could be missing something :) If you're still stuck, see if you can add some more of the surrounding code and what your trying to achieve. Hope this helps.

PS: there is also a }; at the end which is invalid in the code you have given us, probably relevant in the actual application though ;)

Acerb

Any syntax errors are typos. I was trying to only show the relevant portions of the code. No, I am not trying to debug with these tests. These tests are for testing specific events and at this point being synchronous really doesn't seem to be the issue. The tests perform as I intend half the time, but when I refresh the browser the results change. That is what my issue is. – Dollfuss

Can you explain how you get 5 assertions in the second test, when only 3 look to be run synchronously? That might be the cause of the problems. I mocked up a fiddle for the second test here and removed the getReport function. It must be the getReport() call that generates the assertions with calls to the jQuery.fn.on(), any chance we could see what that function does? – Acerb

The whole function call is done synchronously. I updated the code to show the function that is being called. The function makes a deep copy of the form, changes some parameters and triggers the submit event. Which is caught by my mock. The problem is that the test will be passing and when I refresh the page the test will fail. Then with another page refresh pass again. Alternating results with each press of F5. – Dollfuss

I don't think there is any async testing going on here. Rather simply when he re-runs the test it passes on one run and will then fail the very next run. It then alternates between passing and failing for each run. – Photochemistry

Yep just saw the update now, sorry. I understood the refresh issue but though it could be a timing related issue. I've updated the fiddle here and you can view it on it's own page here. In IE, Chrome and firefox it runs without a hitch. I had to write my own jQuery.fn.on() as I don't know what going on in your code, but hopefully it will help narrow things down for you. – Acerb

One final jsfiddle here with the jQuery.fn.on contained within a qunit test. Still can't replicate the behaviour though, sorry. Should help you narrow it down to other code though. – Acerb