Node.js pm2 keeps restarting almost every second

Question

I have deployed an express.js app on a Azure server. I use pm2 for process management.

The issue is pm2 keeps restarting almost every seconds.

staging@Server:/srv/apps/myapp/current$ pm2 list
┌──────────┬────┬──────┬───────┬────────┬─────────┬────────┬─────────────┬──────────┐
│ App name │ id │ mode │ pid   │ status │ restart │ uptime │ memory      │ watching │
├──────────┼────┼──────┼───────┼────────┼─────────┼────────┼─────────────┼──────────┤
│ app      │ 0  │ fork │ 35428 │ online │ 0       │ 0s     │ 20.465 MB   │ disabled │
└──────────┴────┴──────┴───────┴────────┴─────────┴────────┴─────────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app
staging@Server:/srv/apps/myapp/current$ pm2 list
┌──────────┬────┬──────┬───────┬────────┬─────────┬────────┬─────────────┬──────────┐
│ App name │ id │ mode │ pid   │ status │ restart │ uptime │ memory      │ watching │
├──────────┼────┼──────┼───────┼────────┼─────────┼────────┼─────────────┼──────────┤
│ app      │ 0  │ fork │ 35492 │ online │ 7       │ 0s     │ 59.832 MB   │ disabled │
└──────────┴────┴──────┴───────┴────────┴─────────┴────────┴─────────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app
staging@Server:/srv/apps/myapp/current$ pm2 list
┌──────────┬────┬──────┬───────┬────────┬─────────┬────────┬─────────────┬──────────┐
│ App name │ id │ mode │ pid   │ status │ restart │ uptime │ memory      │ watching │
├──────────┼────┼──────┼───────┼────────┼─────────┼────────┼─────────────┼──────────┤
│ app      │ 0  │ fork │ 35557 │ online │ 13      │ 0s     │ 21.816 MB   │ disabled │
└──────────┴────┴──────┴───────┴────────┴─────────┴────────┴─────────────┴──────────┘

~/.pm2/pm2.log

2016-05-10 17:39:34: Starting execution sequence in -fork mode- for app name:start id:0
2016-05-10 17:39:34: App name:start id:0 online
2016-05-10 17:39:35: App [start] with id [0] and pid [3149], exited with code [255] via signal [SIGINT]
2016-05-10 17:39:35: Starting execution sequence in -fork mode- for app name:start id:0
2016-05-10 17:39:35: App name:start id:0 online
2016-05-10 17:39:35: App [start] with id [0] and pid [3158], exited with code [255] via signal [SIGINT]
2016-05-10 17:39:35: Starting execution sequence in -fork mode- for app name:start id:0
2016-05-10 17:39:35: App name:start id:0 online
2016-05-10 17:39:36: App [start] with id [0] and pid [3175], exited with code [255] via signal [SIGINT]
2016-05-10 17:39:36: Starting execution sequence in -fork mode- for app name:start id:0

I am using coffee script in my application. And starting the app using pm2 start app.coffee

package.json

{
  "name": "myapp",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "start": "gulp start-server"
  },
  "dependencies": {
    "bcrypt-nodejs": "0.0.3",
    "body-parser": "~1.13.2",
    "co": "^4.6.0",
    "coffee-script": "^1.10.0",
    "connect-mongo": "^1.1.0",
    "cookie-parser": "~1.3.5",
    "debug": "~2.2.0",
    "express": "~4.13.1",
    "express-session": "^1.13.0",
    "gulp": "^3.9.1",
    "mongoose": "^4.4.14",
    "morgan": "~1.6.1",
    "newrelic": "^1.26.2",
    "passport": "^0.3.2",
    "passport-local": "^1.0.0",
    "pm2": "^1.1.3",
    "pug": "^2.0.0-alpha6",
    "request": "^2.72.0",
    "serve-favicon": "~2.3.0"
  },
  "devDependencies": {
    "shipit-cli": "^1.4.1",
    "shipit-deploy": "^2.1.3",
    "shipit-npm": "^0.2.0",
    "shipit-pm2-nginx": "^0.1.8"
  }
}

I am new to node.js. May be I am not seeing the obvious. Please help me out.

Rolan

not familiar with pm2. can you share any sample project that can be use for reproduce the issue? – Capitulum

start your process by hand, it should show the reason of the problem, fix it, then use pm2. – Sixfold

@XiaominWu Here it is - github.com/resaca/node_debug_app. I am not sure you can reproduce the issue. Because I used the same command on my development machine and no issue there. – Rolan

@mh-cbon Yes, I tried running pm2 start app.coffee from the server without shipit. Still the same. – Rolan

I mean node something.js. BTW, you need to build the coffee script to JS before executing them. – Sixfold

@mh-cbon Thank you so much. You just saved the day. It's again careless me. I didn't have mongodb installed on server. Could you put it as an answer so that I can accept it? – Rolan

@XiaominWu Never mind. It's solved. Thank you for your time. – Rolan

i find it weird that pm2 does not report the startup failure. Did you miss some configuration ? – Sixfold

I actually missed some logs. I only looked in pm2.log. There was ~/.pm2/logs directory and it had app specific logs. – Rolan

good to know ! : ) – Sixfold

Update your PM2. In my case, I had done a node.js upgrade and had to: sudo npm i pm2 --global After this, everything worked without the constant restarting of PM2 – Yonah

Answer 1

pm2 writes application logs to ~/.pm2/logs and pm2 specific logs to pm2.log by default. We need to check both the locations to debug the issue.

One other way to debug application is by starting the application manually, ie., something like npm run start or node path/yo/your/bin.js

It should give you the missing piece of information to fix the problem and move on.

Sixfold

node ./bin/appname solved it for me in Express 4- it gave me the correct errors with my app (filename case problems). I also had the wrong version of Node.js and had to downgrade to solve my specific problem 10.x.x to 8.5.0 (but I can probably fix this for Node.js 10.x.x in the future). – Jameyjami

Answer 2

Check if your app modifies a file in the project folder (such as a log file). A change to any of the files triggers restart if watch flag is enabled.

To prevent this, use a process file and add watch_ignore flag in it.

Here's a documentation on how to use the process file: PM2 - Process File

Cognomen

Thank you that was it for me! This certainly led to some pretty odd behavior. – Navarrette

This drove me like a crazy for almost an hour, thanks – Historiography

You are a lifesaver. – Tripoli

After 2 weeks of debugging blindly, this saved us! – Nippur

Answer 3

We also faced a similar problem where pm2 was restarting a process to start a node.js web application almost every second.

We found that MongoDB was not running, and then the web application would try to connect to the database on start up but would fail. This would prompt pm2 to restart the process over and over, causing a restart every second.

If this is your issue, try to start MongoDB with mongod or mongod --dbpath [your db path]?

Burger

Answer 4

Applicable if you have packaged and started your app with NPM.

I simply had to change the "script" file in the ecosystem.configure.js ( or the json file if you are using ). app.js will not work, I had to replace it with ./bin/www and then it worked.

Pharmacy

Answer 5

Be sure to look at the logs to see what is going wrong (pm2 describe {process} will show you where they are saved). Also, see if you can run the express app without pm2 by stopping the pm2 process and running your app manually (i.e. npm run start).

If you can run the app manually but it doesn't work with pm2, it might be that the app is not being run from the correct directory (you can modify this with the pm2 cwd argument).

Another common issue is that the correct environment variables are not set, so check your json or ecosystem file. You can also look at the environment that pm2 is running with pm2 prettylist.

Cornelia

Answer 6

I know this is kinda late and everything but for anyone scrolling over this, I found n actual solution, after hours of researching.

So I wanted to share this cheatsheet found: https://devhints.io/pm2

pm2 start app.js --no-autorestart

Langsyne

Answer 7

Just ran into this error too. I ran dmesg and that told me my process was getting killed by the Linux kernel, as it was using more memory than I had given the Docker container it was running inside.

Allocating more memory to the container fixed the problem in this case.

Skinflint