Browserstack reports successful even when test fails in Nightwatchjs

Question

I just started using nightwatch with browserstack and I'm noticing that when we get a failed test, nightwatch registers the failure, but browserstack does not. sample test I am using. Also I am using free trial version of BrowserStack.

My question is:

1. Are there any ideas how to tell browserstack when a test run failed ?

From BrowserStack doc:

REST API

It is possible to mark tests as either a pass or a fail, using the following snippet:

var request = require("request"); 
request({
    uri: "https://user:[email protected]/automate/sessions/<session-id>.json",
    method: "PUT",
    form: {
        "status": "completed",
        "reason":""
    }
});

The two potential values for status can either be completed or error. Optionally, a reason can also be passed.

My questions are:

2. How I can get 'session-id' after test execution ?
3. What if I can see "completed" status in dashboard already ?

Goodoh

Answer 1

1. A session on BrowserStack has only three types of statuses: Completed, Error or Timeout. Selenium (and hence, BrowserStack) does not have a way of understanding, if a test has passed or failed. Its by the multiple assertions in your tests that appear on your console, that you infer if a test has passed / failed. These assertions however, do not reach BrowserStack. As you rightly identified, you can use the REST-API, to change the status of the session to 'Error', if you see a failure in your console.

2. I would suggest fetching the session ID of the test as the test is being executed, since fetching the session ID after the test execution is a lengthy process. In Nightwatch, you can fetch session ID as follows:

browser.session(function(session) {
    console.log(session.sessionId);
});

3. Yes, you can certainly change the status of the session once it is completed. That's where the REST-API comes to help!

Directory

Is there an easier way to force fail the test at BrowserStack? Instead of having to meddle with the API, figuring out the build and session ID... – Inexpert

You can mark the session as "Error" only via REST-API. There is no other approach. That said, you don't need the build ID to change the status of a session, only the session ID is enough. Check this link for more details - browserstack.com/automate/rest-api#rest-api-sessions – Directory

Their support told me that I could do driver.getSessionId() to get the id right away. – Inexpert

Depends on what language you are using. If you are using Nightwatch, you can use the method shared above. For Java, it is driver.getSessionId() to get the session ID. – Directory

Answer 2

If you came here searching for a solution in Python, you could use

requests.put(
    "https://api.browserstack.com/automate/sessions/{}.json".format(driver.session_id),
    auth=(USERNAME, ACCESS_KEY),
    json={"status": "failed", "reason": "test failed"})  

Fontanel