Protractor flakiness

Question

I maintain a complex Angular (1.5.x) application that is being E2E tested using Protractor (2.5.x). I am experiencing a problem with this approach, which presents primarily in the way the tests seem flaky. Tests that worked perfectly well in one pull request fail in another. This concerns simple locators, such as by.linkTest(...). I debugged the failing tests and the app is on the correct page, the links are present and accessible.

Has anyone else experienced these consistency problems? Knows of a cause or workaround?

Photoperiod

Answer 1

Just Say No to More End-to-End Tests!

That said, here are the few things you can do to tackle our mutual merciless "flakiness" enemy:

• update to the latest Protractor (currently 4.0.0) which also brings latest selenium and chromedriver with it
• turn off Angular animations

• use dragons browser.wait() with a set of built-in or custom Expected Conditions. This is probably by far the most reliable way to approach the problem. Unfortunately, this is use-case and problem specific, you would need to modify your actual tests in the problematic places. For example, if you need to click an element, wait for it to be clickable:

  var EC = protractor.ExpectedConditions;
  var elm = $("#myid");
  
  browser.wait(EC.elementToBeClickable(elm), 5000);
  elm.click();
  

• maximize the browser window (to avoid random element not visible or not clickable errors). Put this to onPrepare():

  browser.driver.manage().window().maximize();
  

• increase the Protractor and Jasmine timeouts
• slow Protractor down by tweaking the Control Flow (not sure if it works for 4.0.0, please test)
• manually call browser.waitForAngular(); in problematic places. I am not sure why this helps but I've seen reports where it definitely helped to fix a flaky test.
• use the jasmine done() callback in your specs. This may help to, for example, not to start the it() block until done is called in beforeEach()
• return a promise from the onPrepare() function. This usually helps to make sure things are prepared for the test run
• use protractor-flake package that would automatically re-run failed tests. More like a quick workaround to the problem

There are also other problem-specific "tricks" like slow typing into the text box, clicking via JavaScript etc.

Endways

Thanks. It's good to know that I'm not alone in this. Also thanks a lot for the handy collection of tips! – Houseyhousey

You'd only be alone if you never had flakey tests... and we'd all want to know how you did it :) Very nice summary @alecxe! – Sensate

It's just annoying when tests fail randomly, about 1 out of 10 or so. You restart the test and it works. This is not what I call determinism. – Houseyhousey

Answer 2

Yes, I think all of us experienced such flakiness issue.

Actually, the flakiness is quite common issue with any browser automation tool. However, this is supposed to be less in case of Protractor as Protractor has built-in wait consideration which performs actions only after loading the dom properly. But, in few cases you might have to use some explicit waits if you see intermittent failures.

I prefer to use few intelligent wait methods like:

function waitForElementToClickable(locator) {
        var domElement = element(by.css(locator)),
            isClickable = protractor.ExpectedConditions.elementToBeClickable(domElement);

        return browser.wait(isClickable, 2000)
            .then(function () {
                return domElement;
            });
    }

Where 2000 ms is used as timeout, you can make it configurable using a variable.Sometimes I also go with browser.sleep() when none of my intelligent wait works.

Fatigued

Answer 3

It's been my experience that some methods (eg. sendKeys()) do not always fire at the expected time, within the controlFlow() queue, and will cause tests to be flakey. I work around this by specifically adding them to the controlFlow(). Eg:

this.enterText = function(input, text) {
    return browser.controlFlow().execute(function() {
        input.sendKeys(text);
    });
};

Sensate

Answer 4

A workaround that my team has been using is to re-run only failed tests using the plugin protractor-errors. Using this tool, we can identify real failures versus flakey tests within 2-3 runs. To add the plugin, just add a require statement to the bottom of the Protractor config's onPrepare function:

exports.config = {
    ...

    onPrepare: function() {
      require('protractor-errors');
    }
}

You will need to pass these additional parameters when to run your tests with the plugin:

protractor config.js --params.errorsPath 'jasmineReports' --params.currentTime (timestamp) --params.errorRun (true or false)

There is also a cli tool that will handle generating the currentTime if you don't have an easy way to pass in a timestamp.

Levan