Puppeteer waitForSelector on multiple selectors
Asked Answered
D

11

24

I have Puppeteer controlling a website with a lookup form that can either return a result or a "No records found" message. How can I tell which was returned? waitForSelector seems to wait for only one at a time, while waitForNavigation doesn't seem to work because it is returned using Ajax. I am using a try catch, but it is tricky to get right and slows everything way down.

try {
    await page.waitForSelector(SELECTOR1,{timeout:1000}); 
}
catch(err) { 
    await page.waitForSelector(SELECTOR2);
}
Dillondillow answered 20/4, 2018 at 17:15 Comment(0)
D
9

Using Md. Abu Taher's suggestion, I ended up with this:

// One of these SELECTORs should appear, we don't know which
await page.waitForFunction((sel) => { 
    return document.querySelectorAll(sel).length;
},{timeout:10000},SELECTOR1 + ", " + SELECTOR2); 

// Now see which one appeared:
try {
    await page.waitForSelector(SELECTOR1,{timeout:10});
}
catch(err) {
    //check for "not found" 
    let ErrMsg = await page.evaluate((sel) => {
        let element = document.querySelector(sel);
        return element? element.innerHTML: null;
    },SELECTOR2);
    if(ErrMsg){
        //SELECTOR2 found
    }else{
        //Neither found, try adjusting timeouts until you never get this...
    }
};
//SELECTOR1 found
Dillondillow answered 25/4, 2018 at 14:10 Comment(0)
U
33

Making any of the elements exists

You can use querySelectorAll and waitForFunction together to solve this problem. Using all selectors with comma will return all nodes that matches any of the selector.

await page.waitForFunction(() => 
  document.querySelectorAll('Selector1, Selector2, Selector3').length
);

Now this will only return true if there is some element, it won't return which selector matched which elements.

Urinate answered 21/4, 2018 at 20:3 Comment(4)
Interesting... so maybe once I know that one of them is there, I can use my try block with a very short timeout?Dillondillow
Yes. That is true.Urinate
heads-up for all, waitFor is now deprecated and will be discontinued soon (using puppeteer 11.x.x)Rainer
How to replicate in python's pyppeteer package?Solarize
K
23

how about using Promise.race() like something I did in the below code snippet, and don't forget the { visible: true } option in page.waitForSelector() method.

public async enterUsername(username:string) : Promise<void> {
    const un = await Promise.race([
        this.page.waitForSelector(selector_1, { timeout: 4000, visible: true })
        .catch(),
        this.page.waitForSelector(selector_2, { timeout: 4000, visible: true })
        .catch(),
    ]);

    await un.focus();
    await un.type(username);
}
Kentigerma answered 20/5, 2020 at 15:46 Comment(2)
This is a brilliant solution. In fact, if promise.all (or allSettled) is the natural solution for an "AND" logic, promise.race should be considered as the natural solution for "OR".Elastomer
Further extending it to: let res = await Promise.race ( [ frame.waitForSelector( ".selector1 ).then( ()=> { return 1 } ).catch(), frame.waitForSelector( ".selector0" ).then( ()=> { return 0 } ).catch() ]); you can also know which selector triggered.Unfit
D
13

I think the best way to approach this is from a more CSS-based perspective. waitForSelector seems to follow the CSS selector list rules. Which means that you can select multiple CSS elements by just using a comma.

try {    
    await page.waitForSelector('.selector1, .selector2', {timeout: 1000})
} catch (error) {
    // handle error
}
Davilman answered 20/10, 2020 at 19:6 Comment(0)
D
9

Using Md. Abu Taher's suggestion, I ended up with this:

// One of these SELECTORs should appear, we don't know which
await page.waitForFunction((sel) => { 
    return document.querySelectorAll(sel).length;
},{timeout:10000},SELECTOR1 + ", " + SELECTOR2); 

// Now see which one appeared:
try {
    await page.waitForSelector(SELECTOR1,{timeout:10});
}
catch(err) {
    //check for "not found" 
    let ErrMsg = await page.evaluate((sel) => {
        let element = document.querySelector(sel);
        return element? element.innerHTML: null;
    },SELECTOR2);
    if(ErrMsg){
        //SELECTOR2 found
    }else{
        //Neither found, try adjusting timeouts until you never get this...
    }
};
//SELECTOR1 found
Dillondillow answered 25/4, 2018 at 14:10 Comment(0)
D
9

In puppeteer you can simply use multiple selectors separated by coma like this:

const foundElement = await page.waitForSelector('.class_1, .class_2');

The returned element will be an elementHandle of the first element found in the page.

Next if you want to know which element was found you can get the class name like so:

const className = await page.evaluate(el => el.className, foundElement);

in your case a code similar to this should work:

const foundElement = await page.waitForSelector([SELECTOR1,SELECTOR2].join(','));
const responseMsg = await page.evaluate(el => el.innerText, foundElement);
if (responseMsg == "No records found"){ // Your code here }
Dodwell answered 19/2, 2021 at 13:45 Comment(0)
S
6

I had a similar issue and went for this simple solution:

helpers.waitForAnySelector = (page, selectors) => new Promise((resolve, reject) => {
  let hasFound = false
  selectors.forEach(selector => {
    page.waitFor(selector)
      .then(() => {
        if (!hasFound) {
          hasFound = true
          resolve(selector)
        }
      })
      .catch((error) => {
        // console.log('Error while looking up selector ' + selector, error.message)
      })
  })
})

And then to use it:

const selector = await helpers.waitForAnySelector(page, [
  '#inputSmsCode', 
  '#buttonLogOut'
])

if (selector === '#inputSmsCode') {
  // We need to enter the 2FA sms code. 
} else if (selector === '#buttonLogOut') {
  // We successfully logged in
}
Selfhypnosis answered 19/12, 2019 at 21:26 Comment(1)
waitForAnySelector basically rolls your own Promise.race.Azarria
E
2

One step further using Promise.race() by wrapping it and just check index for further logic:

// Typescript
export async function racePromises(promises: Promise<any>[]): Promise<number> {
  const indexedPromises: Array<Promise<number>> = promises.map((promise, index) => new Promise<number>((resolve) => promise.then(() => resolve(index))));
  return Promise.race(indexedPromises);
}
// Javascript
export async function racePromises(promises) {
  const indexedPromises = promises.map((promise, index) => new Promise((resolve) => promise.then(() => resolve(index))));
  return Promise.race(indexedPromises);
}

Usage:

const navOutcome = await racePromises([
  page.waitForSelector('SELECTOR1'),
  page.waitForSelector('SELECTOR2')
]);
if (navigationOutcome === 0) {
  //logic for 'SELECTOR1'
} else if (navigationOutcome === 1) {
  //logic for 'SELECTOR2'
}


Embellish answered 11/10, 2020 at 20:44 Comment(0)
A
1

If you want to wait for the first of multiple selectors and get the matched element(s), you can start with waitForFunction:

const matches = await page.waitForFunction(() => {
  const matches = [...document.querySelectorAll(YOUR_SELECTOR)];
  return matches.length ? matches : null;
});

waitForFunction will return an ElementHandle but not an array of them. If you only need native DOM methods, it's not necessary to get handles. For example, to get text from this array:

const contents = await matches.evaluate(els => els.map(e => e.textContent));

In other words, matches acts a lot like the array passed to $$eval by Puppeteer.

On the other hand, if you do need an array of handles, the following demonstration code makes the conversion and shows the handles being used as normal:

const puppeteer = require("puppeteer"); // ^16.2.0

const html = `
<!DOCTYPE html>
<html>
<head>
<style>
h1 {
  display: none;
}
</style>
</head>
<body>
<script>
setTimeout(() => {

  // add initial batch of 3 elements
  for (let i = 0; i < 3; i++) {
    const h1 = document.createElement("button");
    h1.textContent = \`first batch #\${i + 1}\`;
    h1.addEventListener("click", () => {
      h1.textContent = \`#\${i + 1} clicked\`;
    });
    document.body.appendChild(h1);
  }

  // add another element 1 second later to show it won't appear in the first batch
  setTimeout(() => {
    const h1 = document.createElement("h1");
    h1.textContent = "this won't be found in the first batch";
    document.body.appendChild(h1);
  }, 1000);

}, 3000); // delay before first batch of elements are added
</script>
</body>
</html>
`;

let browser;
(async () => {
  browser = await puppeteer.launch({headless: true});
  const [page] = await browser.pages();
  await page.setContent(html);

  const matches = await page.waitForFunction(() => {
    const matches = [...document.querySelectorAll("button")];
    return matches.length ? matches : null;
  });
  const length = await matches.evaluate(e => e.length);
  const handles = await Promise.all([...Array(length)].map((e, i) =>
    page.evaluateHandle((m, i) => m[i], matches, i)
  ));
  await handles[1].click(); // show that the handles work
  const contents = await matches.evaluate(els => els.map(e => e.textContent));
  console.log(contents);
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close())
;

Unfortunately, it's a bit verbose, but this can be made into a helper.

See also Wait for first visible among multiple elements matching selector if you're interested in integrating the {visible: true} option.

Azarria answered 6/9, 2022 at 5:18 Comment(1)
🚀 Excellent solution for matching multiple selectors.Footloose
A
0

Combining some elements from above into a helper method, I've built a command that allows me to create multiple possible selector outcomes and have the first to resolve be handled.

/**
 * @typedef {import('puppeteer').ElementHandle} PuppeteerElementHandle
 * @typedef {import('puppeteer').Page} PuppeteerPage
 */

/** Description of the function
  @callback OutcomeHandler
  @async
  @param {PuppeteerElementHandle} element matched element
  @returns {Promise<*>} can return anything, will be sent to handlePossibleOutcomes
*/

/**
 * @typedef {Object} PossibleOutcome
 * @property {string} selector The selector to trigger this outcome
 * @property {OutcomeHandler} handler handler will be called if selector is present
 */

/**
 * Waits for a number of selectors (Outcomes) on a Puppeteer page, and calls the handler on first to appear,
 * Outcome Handlers should be ordered by preference, as if multiple are present, only the first occuring handler
 * will be called.
 * @param {PuppeteerPage} page Puppeteer page object
 * @param {[PossibleOutcome]} outcomes each possible selector, and the handler you'd like called.
 * @returns {Promise<*>} returns the result from outcome handler
 */
async function handlePossibleOutcomes(page, outcomes)
{
  var outcomeSelectors = outcomes.map(outcome => {
    return outcome.selector;
  }).join(', ');
  return page.waitFor(outcomeSelectors)
  .then(_ => {
    let awaitables = [];
    outcomes.forEach(outcome => {
      let await = page.$(outcome.selector)
      .then(element => {
        if (element) {
          return [outcome, element];
        }
        return null;
      });
      awaitables.push(await);
    });
    return Promise.all(awaitables);
  })
  .then(checked => {
    let found = null;
    checked.forEach(check => {
      if(!check) return;
      if(found) return;
      let outcome = check[0];
      let element = check[1];
      let p = outcome.handler(element);
      found = p;
    });
    return found;
  });
}

To use it, you just have to call and provide an array of Possible Outcomes and their selectors / handlers:

 await handlePossibleOutcomes(page, [
    {
      selector: '#headerNavUserButton',
      handler: element => {
        console.log('Logged in',element);
        loggedIn = true;
        return true;
      }
    },
    {
      selector: '#email-login-password_error',
      handler: element => {
        console.log('password error',element);
        return false;
      }
    }
  ]).then(result => {
    if (result) {
      console.log('Logged in!',result);
    } else {
      console.log('Failed :(');
    }
  })
Antimonyl answered 21/8, 2019 at 0:33 Comment(0)
J
0

I just started with Puppeteer, and have encountered the same issue, therefore I wanted to make a custom function which fulfills the same use-case.

The function goes as follows:

async function waitForMySelectors(selectors, page){
    for (let i = 0; i < selectors.length; i++) {
        await page.waitForSelector(selectors[i]);
    }
}

The first parameter in the function recieves an array of selectors, the second parameter is the page that we're inside to preform the waiting process with.

calling the function as the example below:

var SelectorsArray = ['#username', '#password'];
await waitForMySelectors(SelectorsArray, page);

though I have not preformed any tests on it yet, it seems functional.

Janettejaneva answered 28/2, 2022 at 23:44 Comment(0)
R
-1

Puppeteer methods might throw errors if they are unable to fufill a request. For example, page.waitForSelector(selector[, options]) might fail if the selector doesn't match any nodes during the given timeframe.

For certain types of errors Puppeteer uses specific error classes. These classes are available via require('puppeteer/Errors').

List of supported classes:

TimeoutError

An example of handling a timeout error:

const {TimeoutError} = require('puppeteer/Errors');

// ...

try {
  await page.waitForSelector('.foo');
} catch (e) {
  if (e instanceof TimeoutError) {
    // Do something if this is a timeout.
  }
}
Rameau answered 17/8, 2018 at 15:47 Comment(0)

© 2022 - 2025 — McMap. All rights reserved.