What I'm doing
Edit: I created a repo with a simplified version of my problem reproducing the issue.
I'm trying to set up automated frontend testings with browserstack, selenium-webdriver and tape.
The idea is to define multiple browsers and devices which have to be tested one after another with X amount of given tests. In the example below I define only one test and two browsers on OSX.
In order to define the browsers only once and handle tests I created a repo test-runner which should be added as dev-dependency to the repos which need to be tested on the given devices and browsers.
The test-runner gets all needed tests passed, starts the first browser, runs the tests on that browser and once all tests are done the browser is closed quit() and the next browser gets started and tests again.
test-runner
/index.js
const webdriver = require( 'selenium-webdriver' )
// ---
// default browser configs
// ---
const defaults = {
"os" : "OS X",
"os_version" : "Mojave",
"resolution" : "1024x768",
"browserstack.user" : "username",
"browserstack.key" : "key",
"browserstack.console": "errors",
"browserstack.local" : "true",
"project" : "element"
}
// ---
// browsers to test
// ---
const browsers = [
{
"browserName" : "Chrome",
"browser_version" : "41.0"
},
{
"browserName" : "Safari",
"browser_version" : "10.0",
"os_version" : "Sierra"
}
]
module.exports = ( tests, url ) => {
// ---
// Asynchronous forEach loop
// helper function
// ---
async function asyncForEach(array, callback) {
for (let index = 0; index < array.length; index++) {
await callback(array[index], index, array)
}
}
// ---
// runner
// ---
const run = async () => {
// ---
// Iterate through all browsers and run the tests on them
// ---
await asyncForEach( browsers, async ( b ) => {
// ---
// Merge default configs with current browser
// ---
const capabilities = Object.assign( {}, defaults, b )
// ---
// Start and connect to remote browser
// ---
console.info( '-- Starting remote browser hang on --', capabilities.browserName )
const browser = await new webdriver.Builder().
usingServer( 'http://hub-cloud.browserstack.com/wd/hub' ).
withCapabilities( capabilities ).
build()
// ---
// Navigate to page which needs to be checked (url)
// ---
console.log('-- Navigate to URL --')
await browser.get( url )
// ---
// Run the tests asynchronously
// ---
console.log( '-- Run tests --- ' )
await asyncForEach( tests, async ( test ) => {
await test( browser, url, capabilities, webdriver )
} )
// ---
// Quit the remote browser when all tests for this browser are done
// and move on to next browser
// Important: if the browser is quit before the tests are done
// the test will throw an error beacause there is no connection
// anymore to the browser session
// ---
browser.quit()
} )
}
// ---
// Start the tests
// ---
run()
}
If you're wondering how this asyncForEach function works I got it from here.
my-repo
/test/front/index.js
const testRunner = require( 'test-runner' )
const url = ( process.env.NODE_ENV == 'development' ) ? 'http://localhost:8888/element/...' : 'https://staging-url/element/...'
// tests to run
const tests = [
require('./test.js')
]
testRunner( tests, url )
/test/front/test.js
const tape = require( 'tape' )
module.exports = async ( browser, url, capabilities, driver ) => {
return new Promise( resolve => {
tape( `Frontend test ${capabilities.browserName} ${capabilities.browser_version}`, async ( t ) => {
const myButton = await browser.wait( driver.until.elementLocated( driver.By.css( 'my-button:first-of-type' ) ) )
myButton.click()
const marked = await myButton.getAttribute( 'marked' )
t.ok(marked == "true", 'Button marked')
//---
// Test should end now
//---
t.end()
resolve()
} )
})
}
/package.json
{
...
"scripts": {
"test": "NODE_ENV=development node test/front/ | tap-spec",
"travis": "NODE_ENV=travis node test/front/ | tap-spec"
}
...
}
When I want to run the tests I execute npm run test in my-repo
Remember, that we have only one test (but could also be multiple tests) and two browsers defined so the behaviour should be:
- Start browser 1 and navigate(Chrome)
- One test on browser 1 (Chrome)
- Close browser 1 (Chrome)
- Start browser 2 and navigate (Safari)
- One test on browser 2 (Safari)
- Close browser 2 (Safari)
- done
The Problem
The asynchronous stuff seems to be working just fine, the browsers are started one after another as intended. The problem is, that the first test does not finish even when i call t.end() and I don't get to the second test (fails right after 4.).
What I tried
I tried using t.pass() and also running the CLI with NODE_ENV=development tape test/front/ | tap-spec but it didn't help.
I also noticed, that when I don't resolve() in test.js the test ends just fine but of course I don't get to the next test then.
I also tried to adapt my code like the solution from this issue but didn't manage to get it work.
Meanwhile I also opened an issue on tapes github page.
So I hope the question is not too much of a pain to read and any help would be greatly appreciated.