PM2 keeps getting killed every 90 seconds on centos 8

Question

I just installed CentOS 8 and added nodejs (tried v12 & v14) And then I installed pm2 using npm install pm2@latest (so at the time of posting it uses v4.4.0). I did try an older version (v3.5.0), but it does the exact same thing.

and after pm2 got installed, i ran the command "pm2 startup"

after a restart, pm2 does start, but gets killed after 90 seconds and then restarts giving this message

"pm2 has been killed by signal, dumping process list before exit..."

First, I thought it was because of my app (the one that pm2 is supposed to manage), but i removed it from pm2, so it's practically empty, but it does the same thing

Garrot

I'm seeing the same thing on RHEL 8.2, node v12.16.1. Every 90 seconds pm2 has been killed by signal, dumping process list before exit... Have you found any solution? I don't see anything useful in my logs. I have the same versions and setup running in another environment, but I don't see the issue there. But every 90 seconds, restart. – Wisniewski

I have a similar issue. It isn't always 90 seconds, sometimes it is 60 seconds and sometimes 120 seconds. I did find that pm2 tests for memory usage every 30 seconds which seems to be related but haven't found anything yet. – Mostly

Have you found anything? The same is happening for me, exactly 90 seconds every time. Whether I'm just running one API or multiple, the whole pm2 process gets restarted.. – Chare

Yes, please read below (3 options). Use either the second or the third for now. – Garrot

Answer 1

Running the following command as root worked for me:

pm2 update

Insinuation

Could you provide more details on what this command does? – Carraway

It updates pm2 itself. – Insinuation

I didn't use sudo and it worked for me (Oracle Linux 8). However, I'm wondering if this needs to be run each time the server restarts and the service begins. Has anybody encountered that behavior or was it just fixed for good at that point? – Allie

seems like not happen anymore after I update – Interne

Answer 2

I did not disable SE Linux (I think it's not safe to disable it), but the following method helped me:

Edit file: /etc/systemd/system/pm2-root.service

1. Add new line: Environment=PM2_PID_FILE_PATH=/run/pm2.pid

2. And replace: PIDFile=/root/.pm2/pm2.pid to: PIDFile=/run/pm2.pid

Versions:

• CentOS 8.3.2011
• Node.js 14.16.0
• NPM 7.7.5
• PM2 4.5.5

Original answer. Thanks Alec!

Parvati

I had the same problem in Fedora with a pm2 instantiated for a user different to root. To avoid changing permissions I've used the /tmp/ folder instead of the /run/ folder – Mutt

Worked for us on Rocky Linux too, thanks :) – Tjaden

This worked to me too, of course using /tmp/ instead of /run/ – Proctor

Answer 3

Later update. For those who are facing the same issues. It's an issue related to SE Linux. Known workarounds (the ones I discovered).

1. Disabling SE Linux (obviously, not recommended)

2. go to /etc/systemd/system/pm2-root.service - comment PIDFile=... (add a # in front of that line)

3. Audit and trace - use following commands:

    # dnf install policycoreutils-python-utils setroubleshoot-server -y
    # journalctl -f
   

   At ths point, you should see the solution in the output (the log) it should be something like:

   # ausearch -c 'systemd' --raw | audit2allow -M my-systemd
   # semodule -i my-systemd.pp
   

   You need to do the last step (ausearch... and semodule...) twice - I did it once, restarted the machine and noticed the same issue after 90 seconds. But if you read the log carefully, you will notice that the issue seems to be outputed twice. (looks the same). Probably two things are trying to write to that file (pm2-root.service).

Still waiting for the perfect solution (done by the person that really knows how to fix this in a proper manner), but for those that have this issue, any of these options seem to work just fine.

Garrot

github.com/DataDog/datadog-agent/issues/5207 – Ardeb

I think it's not safe to disable SE Linux – Parvati

Option 2 seems to have worked for me. Can anyone explain how this works? – Hippodrome

Used option 2 then rebooted the server. Problem solved. Any security concerns? idk how this works either. I'm on Ubuntu though btw – Listel

I had to apply option 2 plus the following steps: Step 1: stop pm2 using systemctl / Step 2: sudo npm i -g pm2@latest / Step 3: sudo systemctl daemon-reload / Step 4: start pm2 again using systemctl – Planimetry

Thank you! Option 2 worked for me too on ubuntu. After commenting the line, i did "pm2 update" and then it was fine. – Dna

Option 2 worked for me – Nahuatl

Answer 4

I had the same issue and I tried several solutions online but none worked for me.

However, I completely removed pm2, restarted the server, and reinstalled pm2 and that does it for me.

1- Stop and remove pm2

pm2 kill
sudo npm remove pm2 -g

2- Restart the server

sudo reboot

3- Log in again, then reinstall pm2

sudo npm install -g pm2

Suburbanite

You might as well remove this because it has nothing to do with Ubuntu. It's a SELinux issue, which doesn't come installed with Ubuntu. – Garrot

@shakirah typo in the last step: "pm", need "pm2" – Parvati

This seems to solve the issue for me. Thanks :) – Poppycock

solved for me, but I had to do: "pm2 update" too – Archuleta

running on ubuntu server, this also solved for me. sudo service pm2-ubuntu status would state that a PID file could not be read. after this reinstall this error is no longer shown – Orgy

Answer 5

I've had this problem (on Debian), when for some reason two "PM2 God Daemon" processes (not threads) were launched, so they conflicting with each other.

Killing one of them solved the issue.

Littlest