Loop through list of clickable elements and write out the html to respective files

Question

I'm using jQuery to get a list of elements that contain certain key words. I'm able to get the list of elements but I don't know how to loop through each element, click on its child element and download the newly loaded page. Here's the casperjs code I have so far:

var casper = require('casper').create({
    clientScripts: ["/var/www/html/project/public/js/jquery-3.3.1.min.js"]
});

var fs = require('fs');

casper.start('https://m.1xbet.co.ke/en/line/Football/', function () {
    var links = casper.evaluate(function () {
        $.expr[":"].contains = $.expr.createPseudo(function (arg) {
            return function (elem) {
                return $(elem).text().toUpperCase().indexOf(arg.toUpperCase()) >= 0;
            };
        });
        return $("#events-betting").find("li.events__item_head:contains(World cup)");
    });

    var date = new Date(), year = date.getFullYear(), month = date.getMonth() + 1, day = date.getDate();
    var folderName = year + '-' + month + '-' + day;

    // loop would go here to save each file
    var path = "destination/" + folderName + "/1xbet/worldcup-1";
    fs.write(path + ".html", this.getHTML(), "w");

});

casper.run();

I'd like to click on the individual items on the links object - they aren't anchor tags but rather they are clickable divs with inline javascript listening for a click.

The goal is to click on the div that has certain text I'm interested in, then once clicked, I can either choose to scrape the HTML and save it in a file or get the current url; either will be fine for my purposes. Since there could be multiple divs with the desired text, I'd like for a way to loop through each and do perform the same operation.

This is an example of the page I'm interested in:

https://m.1xbet.co.ke/en/line/Football/

The parent element in this case is: #events-betting and nested is a list of li tags with clickable divs.

Regular

Possible duplicate of CasperJS - How to open up all links in an array of links – Restaurant

@Restaurant Still haven't found a way forward. Any help would be appreciated. Added a couple of lines for further clarity. – Regular

I'd suggest exploring their API. Look at this nice and clear data A request like this is generated each time you click on a menu item. champs correlates with data-champ from a menu item. – Restaurant

Thank you for the response. I would consider the API but this is just one of a couple of sites I'm scraping and not all have APIs. This should help for this particular site though. – Regular

Answer 1

I can either choose to scrape the HTML and save it in a file or get the current url

Of course the solution is very specific to this exact site, but then again it is quite normal when doing web scraping.

casper.start('https://m.1xbet.co.ke/en/line/Football/', function () {

  var links = casper.evaluate(function () {

    $.expr[":"].contains = $.expr.createPseudo(function (arg) {
      return function (elem) {
        return $(elem).text().toUpperCase().indexOf(arg.toUpperCase()) >= 0;
      };
    });

    var links = [];
    // Better to scrpape .events__title as it contains data-href attribute
    $("#events-betting").find(".events__title:contains(World cup)").each(function (i, item) {
      var lastPartOfurl = item.getAttribute("data-href");
      lastPartOfurl = lastPartOfurl.split("/");
      links.push("https://m.1xbet.co.ke/en/line/Football/" + item.getAttribute("data-champ") + "-" + lastPartOfurl[1]+'/');
    })

    return links;
  });

  console.log(links);
});

The result:

https://m.1xbet.co.ke/en/line/Football/1536237-FIFA-World-Cup-2018/,https://m.1xbet.co.ke/en/line/Football/1204917-FIFA-World-Cup-2018-Winner/,https://m.1xbet.co.ke/en/line/Football/1518431-FIFA-World-Cup-2018-Special-bets/,https://m.1xbet.co.ke/en/line/Football/1706515-FIFA-World-Cup-2018-Teams-Statistics-Group-Stage/

Restaurant

Thank you @Restaurant It works for this and similar sites. You've really helped me out. Some sites don't have the data attributes that allow me to construct similar outputs but at least I'm halfway there with this solution. Once again, really appreciate the input. – Regular