How to use Selenium Webdriver on Heroku?

Question

I am developing a Node.js app, and I use Selenium Webdriver on it for scraping purposes. However, when I deploy on Heroku, Selenium doesn't work. How can I make Selenium work on Heroku?

Dictograph

Hello man, have you found the solution for this? I'm also using Node on the server and using Angular on frontend, everything works locally but after I deployed it on heroku, selenium doesn't work – Tournai

check my answer here: https://mcmap.net/q/831209/-how-to-run-selenium-webdriver-on-heroku-with-node-js-firefox-or-chrome – Amieeamiel

Answer 1

Below is a javaScript sample code using selenium-webdriver npm package with chrome browser.

const webdriver = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');

let options = new chrome.Options();
//Below arguments are critical for Heroku deployment
options.addArguments("--headless");
options.addArguments("--disable-gpu");
options.addArguments("--no-sandbox");

let driver = new webdriver.Builder()
  .forBrowser('chrome')
  .setChromeOptions(options)
  .build();

driver.get('http://www.google.com');
driver.quit();

Before you are ready to deploy, you would need to add two buildpacks to Heroku.

• Using Heroku buildpacks command:

$ heroku buildpacks:add --index 1 https://github.com/heroku/heroku-buildpack-chromedriver
$ heroku buildpacks:add --index 2 https://github.com/heroku/heroku-buildpack-google-chrome

or

• Config in Heroku dashboard:
  Settings -> Add buildpacks -> https://github.com/heroku/heroku-buildpack-chromedriver -> Save changes
  Settings -> Add buildpacks -> https://github.com/heroku/heroku-buildpack-google-chrome -> Save changes
  

Denti

Hi folks, so with this setup is it possible to disable w3c mode? – Ebneter

Answer 2

I was able to get the Selenium Webdriver working on Node/Heroku using PhantomJs as the headless browser. I installed the PhantomJs buildpack to my Heroku app and it just worked. I struggled to get Chrome and Firefox drivers working on Heroku... I wrote a blog with the steps and code I used to get it working:

http://www.viderman.com/2017/05/selenium-on-heroku.html

Nunes

Please include the steps and code in your answer. Answers should not rely on external links for the majority of their information. – Santee

Same could be said about the blog post :) External references have a way of being deleted. – Demigod

Answer 3

Pasting my answer from here: https://mcmap.net/q/831209/-how-to-run-selenium-webdriver-on-heroku-with-node-js-firefox-or-chrome

---

Note: I am using React, Express, and Selenium with chrome

Step 1: Create a new Heroku app.

Step 2: From your terminal, login to heroku using heroku login

step 3: Once you're logged in, cd to your project directory and set its remote to your heroku app. heroku git:remote -a YOUR-HEROKU-APP-NAME

step 4: Run all the following commands in your terminal

heroku buildpacks:add https://github.com/heroku/heroku-buildpack-chromedriver
heroku buildpacks:add https://github.com/heroku/heroku-buildpack-google-chrome
heroku config:set CHROME_DRIVER_PATH=/app/.chromedriver/bin/chromedriver
heroku config:set CHROME_BINARY_PATH=/app/.apt/opt/google/chrome/chrome

step 5: Login to heroku from your browser and navigate to your app. Go to settings and under buildpacks, add heroku/nodejs

step 6: This is how my index.js looks like. Note: my express entry point is inside root-dir/server/index.js and my react files are inside root-dir/client/

const express = require('express');
const app = express();
const path = require('path');

// Serve static files from the React app. 
app.use(express.static(path.join(__dirname, '..', 'client/build')));


app.get('/api', async (req, res) => {
    const webdriver = require('selenium-webdriver');
    require('chromedriver');
    const chrome = require('selenium-webdriver/chrome');

    let options = new chrome.Options();
    options.setChromeBinaryPath(process.env.CHROME_BINARY_PATH);
    let serviceBuilder = new chrome.ServiceBuilder(process.env.CHROME_DRIVER_PATH);
    
    //Don't forget to add these for heroku
    options.addArguments("--headless");
    options.addArguments("--disable-gpu");
    options.addArguments("--no-sandbox");
  

    let driver = new webdriver.Builder()
        .forBrowser('chrome')
        .setChromeOptions(options)
        .setChromeService(serviceBuilder)
        .build();

    await driver.get('http://www.google.com');
    res.send(await driver.getTitle());
});

app.get('*', (req, res) => {
    res.sendFile(path.join(__dirname, '..', 'client/build/index.html'));
});

const port = process.env.PORT || 5000;
app.listen(port, () => {
    console.log(`listening to port ${port} now...`);
});

step 7 (if you are using React): Now inside your package.json in root-dir/, add this

"scripts": {
...
"heroku-postbuild": "cd client && npm install && npm run build"
}

step 8 (if you are using react): inside your package.json in root-dir/client/ (i.e: package.json for react app), add the following line:

  "proxy": "http://localhost:5000/",

step 8: (if you're using react): inside root-dir/client/src/, create a new file called setupProxy.js and paste the following code:

const proxy = require("http-proxy-middleware");

module.exports = function(app) {
    app.use(proxy('/api', { target: `http://localhost:${process.env.PORT || 5000}/`}));
};

step 9: Now, you are ready for deployment. Make sure you have the following packages installed: express, selenium-webdriver, and chromedriver

step 10: now push it to heroku

git add .
git commit -m "my app"
git push heroku master

Amieeamiel