How to improve puppeteer startup performance during tests

Question

I've written a small crawler with the help of Puppeteer. Now I'm facing the challenge that my tests are rather slowly (> 3 seconds for each test). I've been able to track it down to the launch function of Puppeteer and the usage of Istanbul/nyc.

• If I run the test just with mocha, the tests are finished under 400 ms.
• But if I additionally use nyc, the duration of the tests exceeds 3000 ms

All that I'm using is

'use strict';
const puppeteer = require('puppeteer');


module.exports = async function startBrowser() {
  const options = {
    args: [
      // '--no-sandbox',
      // '--disable-setuid-sandbox',
      // '--disable-dev-shm-usage',
      // '--disable-accelerated-2d-canvas',
      // '--disable-gpu'
    ],
    headless: false // true
  };

  return await puppeteer.launch(options);
};

Here is the test I'm using:

'use strict';
/* global describe: false, before: false, it: false,
    beforeEach: false, afterEach: false, after: false, window: false, document: false */

const assert = require('assert').strict;
const startBrowser = require('../');
const util = require('util');



describe('Puppeteer', function() {
  let pageManager;

  it('start the browser', async function() {
    this.timeout(10000);

    console.time('startBrowser');
    const browser = await startBrowser();
    console.timeEnd('startBrowser');
    assert(browser);

    console.time('closeBrowser');
    await browser.close();
    console.timeEnd('closeBrowser');
  });

});

I've created a repository with this code and test here. nyc _mocha ./test/*.test.js runs in ~3500ms, mocha ./test/*.test.js takes only 130ms.

What I've tried so far:

• different combination of include/exclude nyc flags
• updating to latest versions of Puppeteer, nyc and mocha
• removing my Puppeteer arguments
• searching for Puppeteer & Istanbul related issues (with not much success)
• trying headless: true
• bypassing all proxies, see this puppeteer issue

---

What can I do to have tests with coverage be as fast as the tests alone?

Using:

• Ubuntu 19.04
• node.js 10.15.3

Absolute

make headless: true than try – Nappe

unfortunately the behavior is the same with headless: true – Absolute

i am not sure but this link could fix take a look, and please check you systems performance [ github.com/GoogleChrome/puppeteer/issues/1846 ] – Nappe

Thank you for your efforts! I tried the direct proxy server, sadly without success. Systems performance is fine, it is fast if I don't use code coverage. I'm interested if you experience the same problem as I do: There is a repo to reproduce it in under a minute => github.com/seasick/puppeteer-slow-nyc – Absolute

With mocha your browser is already launched. With puppeteer you are launching a new browser virtually. Maybe if they introduced a state-save feature to it then subsequent tests might run faster but ultimately it's probably going to stay slow. That said, it's still way better than manually clicking through my project every time I (or anyone else) do work. – Foucault

Answer 1

I've started to debug Puppeteer and these are my findings:

• Puppeteer is unsurprisingly using child_process.spawn() to spawn a new browser
• nyc is using spawn-wrap for such child processes
• spawn-wrap is reading the whole executable (./node_modules/puppeteer/.local-chromium/linux-686378/chrome-linux/chrome) into memory with fs.readFileSync which is taking an unusually long time to finish

spawn-wraps README delivers some kind of an explanation:

The initial wrap call uses synchronous I/O. Probably you should not be using this script in any production environments anyway. Also, this will slow down child process execution by a lot, since we're adding a few layers of indirection.

For me personally the answer is that I cannot get the same performance for running tests with and without code coverage as long as I use nyc/istanbul.

I've given c8 a shot, and the performance is nearly the same and I can still have code coverage.

Absolute

Answer 2

Please try this also.

'use strict'
const puppeteer = require('puppeteer')

module.exports = async function startBrowser() {
  const options = {
    args: [
      '--no-sandbox',
      '--disable-setuid-sandbox',
      '--disable-dev-shm-usage',
      '--disable-accelerated-2d-canvas',
      '--no-first-run',
      '--no-zygote',
      '--single-process',
      '--disable-gpu'
    ],
    headless: true
  }

  return await puppeteer.launch(options)
}

Draughts

unfortunately this does not change the launch performance for me – Absolute

i've add a screenshot of your test running in my terminal – Draughts

and you are saying that with your changes, the time is nearly the same? In your screenshot, the tests are faster, but the command with coverage (npm run test) is still taking 6x the time its running without (npm run test_mocha) – Absolute

let me try this as well on my i3 with 8gb of ram ... and let's see the result. :D – Yoshi

--single-process Is apparently now working in Windows, saving me ~500ms on browser.newPage – Barri

Thanks for validating this flag. @JuanjoMartinez – Draughts

Answer 3

1. ./Chromium --headless --disable-gpu --remote-debugging-port=9222 --devtools=false
2. browser = await puppeteer.connect({ browserWSEndpoint: 'ws://127.0.0.1:9222/devtools/browser/........xxxxxx..', });
3. use express to hold it

PS: But ,I can't specify the browserWSEndpoint the url change every time Chromium --headless restart

Thiouracil