How to run nightmare.js on google appengine for node.js

Question

There is famous issue with nightmare and electron not running on headless linux servers. The official electron docs suggest to use xvfb in order to fake the display. They suggest to use this .yml file for travis.

addons:
  apt:
    packages:
      - xvfb

install:
  - export DISPLAY=':99.0'
  - Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &

Question

How can I use the above piece of code into app.yaml file of google appengine for node.js. I have tried to use this as it is, but glcoud throws an error that addon is invalid command. The gap official docs don't have any similar command.

Any suggestions how we can run nightmare and electron on google appengine for node.js..?

Caridadcarie

Answer 1

There are two parts related to this question:

1. Running chromium (what electron and, in turn, nightmare "uses") headlessly on linux.
2. Install/Use xvfb to run chromium on app engine.

Part 1)

You need xvfb.

Xvfb (Virtual Framebuffer) is just a program that, from wiki: "is a display server implementing the X11 display server protocol. In contrast to other display servers, Xvfb performs all graphical operations in memory without showing any screen output."

Which is what you need to run a browser without a screen output.

First, install all the xvfb related packages to run it on linux.

apt-get install -y \ xvfb \ x11-xkb-utils \ xfonts-100dpi \ xfonts-75dpi \ xfonts-scalable \ xfonts-cyrillic \ x11-apps \ clang \ libdbus-1-dev \ libgtk2.0-dev \ libnotify-dev \ libgnome-keyring-dev \ libgconf2-dev \ libasound2-dev \ libcap-dev \ libcups2-dev \ libxtst-dev \ libxss1 \ libnss3-dev \ gcc-multilib \ g++-multilib

So with xvfb installed you need to create a virtual xvfb screen and export an environment variable called DISPLAY that points to it. Chromium in Electron will automatically look for $DISPLAY.

The above can be done more easily. Here are two options:

• Calling the program with linux cli (ignore the xvfb warnings if the nightmare script runs fine):

  • xvfb-run -a node main.js. Or...

  • If using rendering related functionality like taking screenshots: xvfb-run -a --server-args="-screen 0 1280x1028x24 -ac +extension GLX +extension RANDR +render" node app.js. Google the xvfb options to adjust to your taste.

• Programmatically: using xvfb npm package

From this point on you should be able to run nightmare on linux.

Part 2)

Nodejs on app engine is ran via the flexible environment. Meaning, through docker containers.

From GAE nodejs runtime: "If your application requires additional operating-system-level dependencies, you will need to use a custom runtime based on this runtime to install the appropriate packages."

Docker is a whole separate topic, but in order to do the above with app engine you have two options as far as I know:

1. Extending the runtime

2. Use GAE with a custom runtime from scratch.

Either way, basically what you would need to do is install the xvfb related packages defining them in the dockerfile and that should do the trick.

Good luck!

Important Notes:

1. The above apt-get packages depend on the availability regarding the linux distro (the above code works on ubuntu and debian). For example, with the specified set of packages and at the time of this post, it will work with GAE's flexible environment since it is based on debian jessie and won't work on linux alpine.

2. Chromium needs a minimum dev/shm allocation to run well. For example, on heroku it is fixed to 5mb -and there is no way to change it. Chromium will crash after a few nightmare actions. So chromium won't work on any heroku's dynos of any size. In docker it is set to 64mb, so depending on the complexity of your script you will do fine or need to adjust it. In plain linux installations, dev/shm is normally set to half of the total available memory. So in a 512mb environment, dev/shm will be set to 256mb and nightmare will most likely run fine.

Tyrannicide

It's worth noting that you CANNOT change /dev/shm on appengine. – Anastasius

Answer 2

Thanks to @rickmed for his thorough answer! It helped me start getting a handle on using xvfb in this context. (https://mcmap.net/q/1271303/-how-to-run-nightmare-js-on-google-appengine-for-node-js)

I am using Nightmare to generate PDF's from an endpoint. My local dev is done on OSX and I got to this issue while trying to get it to work on Google App Engine. I got it working initially with rickmed's answer and have since figured out another way that avoids the custom Dockerfile/runtime. I thought I would share it here.

I am not using a custom Dockerfile and allow gcloud to generate one for me during deploy. My yaml file uses runtime: nodejs. For my simple use of Nightmare I am able to add a preinstall script to my package.json and update the start script. That's all I need to get nightmare working on GAE. Here are the relevant lines from my package.json:

{
  "scripts": {
    "preinstall": "apt-get update && apt-get install -y libgtk2.0-0 libgconf-2-4 libasound2 libxtst6 libxss1 libnss3 xvfb",
    "start": "xvfb-run -a node build/server/index",
    ...
  },
  ...
}

I pulled the simplified set of apt-get installed packages from otaviomedeiros's comment: https://github.com/segmentio/nightmare/issues/224#issuecomment-225887320

I got the idea from Daishi Kato's helpful article: https://medium.com/google-cloud/how-to-use-phantomjs-with-node-js-on-google-app-engine-6f7feaea551#.6eoyvpn93 and this disclaimer is included in the article:

Although the following procedure works well as of writing, that doesn’t mean it will work for a long time. I am not even sure if it is recommended. Please understand the risk.

So take this for what it is, and hopefully it will help someone!

Conservative