Puppeteer | Wait for all JavaScript is executed

Question

I try to take screenshots from multiple pages, which should be fully loaded (including lazy loaded images) for later comparison.

I found the lazyimages_without_scroll_events.js example which helps a lot.

With the following code the screenshots are looking fine, but there is some major issue.

async function takeScreenshot(browser, viewport, route) {
  return browser.newPage().then(async (page) => {
    const fileName = `${viewport.directory}/${getFilename(route)}`;

    await page.setViewport({
      width: viewport.width,
      height: 500,
    });
    await page.goto(
        `${config.server.master}${route}.html`,
        {
          waitUntil: 'networkidle0',
        }
    );
    await page.evaluate(() => {
      /* global document,requestAnimationFrame */
      let lastScrollTop = document.scrollingElement.scrollTop;

      // Scroll to bottom of page until we can't scroll anymore.
      const scroll = () => {
        document.scrollingElement.scrollTop += 100;
        if (document.scrollingElement.scrollTop !== lastScrollTop) {
          lastScrollTop = document.scrollingElement.scrollTop;
          requestAnimationFrame(scroll);
        }
      };
      scroll();
    });
    await page.waitFor(5000);
    await page.screenshot({
      path: `screenshots/master/${fileName}.png`,
      fullPage: true,
    });

    await page.close();
    console.log(`Viewport "${viewport.name}", Route "${route}"`);
  });
}

Issue: Even with higher values for page.waitFor() (timeout), sometimes not the all of the frontend related JavaScripts on the pages were fully executed.

For some older pages where some JavaScript could change the frontend. F.e. in one legacy case a jQuery.matchHeight.

Best case: In an ideal world Puppeteer would wait till all JavaScript is evaluated and executed. Is something like this possible?

---

EDIT
I could improve the script slightly with the help from cody-g.

function jQueryMatchHeightIsProcessed() {
  return Array.from($('.match-height')).every((element) => {
    return element.style.height !== '';
  });
}

// Within takeScreenshot() after page.waitFor()
await page.waitForFunction(jQueryMatchHeightIsProcessed, {timeout: 0});

... but it is far from perfect. It seems I have to find similar solutions for different frontend scripts to really consider everything which happening on the target page.

The main problem with jQuery.matchHeight in my case is that it does process different heights in different runs. Maybe caused by image lazyloading. It seems I have to wait until I can replace it with Flexbox. (^_^)°

Other issues to fix:

Disable animations:

await page.addStyleTag({
  content: `
    * {
      transition: none !important;
      animation: none !important;
    }
  `,
});

Handle slideshows:

function handleSwiperSlideshows() {
  Array.from($('.swiper-container')).forEach((element) => {
    if (typeof element.swiper !== 'undefined') {
      if (element.swiper.autoplaying) {
        element.swiper.stopAutoplay();
        element.swiper.slideTo(0);
      }
    }
  });
}

// Within takeScreenshot() after page.waitFor()
await page.evaluate(handleSwiperSlideshows);

But still not enough. I think it's impossible to visual test these legacy pages.

Heisser

Did you ever find a general-purpose way to accomplish this? – Obsolete

Sadly not. All I did is mentioned in my EDIT. – Heisser

Answer 1

The following waitForFunction might be useful for you, you can use it to wait for any arbitrary function to evaluate to true. If you have access to the page's code you can set the window status and use that to notify puppeteer it is safe to continue, or just rely on some sort of other ready state. Note: this function is a polling function, and re-evaluates at some interval which can be specified.

const watchDog = page.waitForFunction('<your function to evaluate to true>');

E.g.,

const watchDog = page.waitForFunction('window.status === "ready"');
await watchDog;

In your page's code you simply need to set the window.status to ready

To utilize multiple watchdogs in multiple asynchronous files you could do something like

index.js

...import/require file1.js;
...import/require file2.js;
...code...

file1.js:

var file1Flag=false; // global
...code...
file1Flag=true;

file2.js:

var file2Flag=false; // global
...code...
file2Flag=true;

main.js:

const watchDog = page.waitForFunction('file1Flag && file2Flag');
await watchDog;

Oto

Yes it would be possible to add code to the target pages. The difficult part for this solution is to find the right spot for this kind of functionality. There are several scripts which gets asynchronously loaded and executed with RequireJS. There is not the single end point. – Heisser

Right. You could create multiple watchdogs in that case though, correct? I'm not aware of any "idle" flag that you could utilize in this case. – Oto

Would you add an simple example code for the target page, which would be possible to utilize with the watchdog, please? – Heisser

Hmmm, not sure what to add, but I added something. – Oto

Answer 2

async function takeScreenshot(browser, viewport, route) {
  return browser.newPage().then(async (page) => {
    const fileName = `${viewport.directory}/${getFilename(route)}`;

    await page.setViewport({
      width: viewport.width,
      height: 500,
    });
    await page.goto(
        `${config.server.master}${route}.html`,
        {
          waitUntil: 'networkidle0',
        }
    );
    await page.evaluate(() => {
      scroll(0, 99999)
    });
    await page.waitFor(5000);
    await page.screenshot({
      path: `screenshots/master/${fileName}.png`,
      fullPage: true,
    });

    await page.close();
    console.log(`Viewport "${viewport.name}", Route "${route}"`);
  });
}

Austroasiatic

Would you add some explanation, please? So far I can see you altered the scrolling behavior. I'm not sure how this could help in my case. – Heisser