Multiple PhantomJS instances hanging

Question

I am having a problem when running multiple instances of PhantomJS on Ubuntu 14. After a few minutes, the processes become unresponsive.

Brief background: Using PhantomJS 2.0 to render a web page that ultimately gets saved as a PDF using Wkhtmtopdf. PhantomJS is only responsible for loading the page, making ajax requests, and waiting for a response after the PDF is saved on the server. It does not generate the PDF itself. There are hundreds of web pages that need to be generated into PDF, so I want to run as many PhantomJS instances in parallel as the system allows.

Each PhantomJS process is started by a shell script like so: {path to phantomjs} {path to js file} --data {some argument} >> {path to log file} 2>&1 &

The problem occurs after a couple of minutes where I stop getting any output from the PhantomJS processes, and looking at top I can see they are just laying there not doing anything. The JS script has timers that retry to load a page if it takes longer than a minute, and eventually call phantom.exit() if the page can't load / PDF generation fails. So even if something goes wrong, the process should still exit - but it doesn't.

I tried changing the number of PhantomJS instances running in parallel. Tried 20 -> 10 -> 5 -> 3, but it doesn't seem to matter. I can actually get many more jobs execute successfully when maintaining 20 instances at a time.

When running with --debug=true I can see that at some point it gets stuck at [DEBUG] WebPage - updateLoadingProgress: Also going through the output I see several of these warnings: [WARNING] QIODevice::write: device not open which makes me believe that is the source of the problem.

I thought there might be some contention for file resources so I tried without redirecting output to a log file, and not using --local-storage-path, but that didn't help.

As a side note, I have been using PhantomJS for several years now doing the same procedure, only sequentially (run a single PhantomJS process at a time). And although there were a few snags to overcome, it worked great.

Any idea what's causing this? Anyone faced with a similar problem? Any advice on running multiple PhantomJS instances in parallel?

Thanks!

Patricepatrich

I'm having the exact same issues! Did you manage to find a solution? Thanks! – Alaniz

@YuviDroid, I haven't found a solid solution. I was able to avoid the problem by limiting the number of instances running at a any given time, and closing / creating a new page object when retrying a job instead of using the same page. It was a while ago, so I don't remember exactly what made it "stable" but, it took a lot of trial and error. And I would like to be able to run many more instances than I currently am running, so a proper solution would be appreciated. – Patricepatrich

What version of PhantomJS are you using? It may be related to this image rendering bug which is supposed to have been fixed as of 2.1.1 -> code.google.com/p/phantomjs/issues/detail?id=54 – Toomay

Have you considered the possibility it's actually the web server causing the issue? Many servers limit concurrent connections (or even do ip-based throttling) and if phantomjs was exceeding the server's own "requests per whatever" limit then it would have to wait (ie, "hang") until the server released more data. – Vallie

abandon phantomJS and use electron or nightmare github.com/segmentio/nightmare electron.atom.io – Margotmargrave

Answer 1

I faced the exact same issue both locally and on my CI server (which was also Ubuntu). Uninstalling 2.0.0 and upgrading to to 2.1.1 resolved the problem for me.

Schriever

It didn't resolve my issue. On the contrary, it seems there is some nasty bug in version 2.1.1 that keeps single instances hanging forever. I noticed that I wasn't getting the PDFs on time, and when I ran top I saw there were dozens of phantomjs processes just hanging. I had to downgrade back to version 2.0 – Patricepatrich

Answer 2

I was facing the same issue. Use driver.quit() instead of driver.close(). That solved the issue for me.

Promoter