"No Such Process" consumes GPU memory

Question

When I use nvidia-smi, I found nearly 20GB GPU Memory is missing somewhere (total listed processes took 17745MB, meanwhile Memory-Usage is 37739MB):

Then I use nvitop, you can see No Such Process has actually taken my GPU resources. However, I cannot kill this PID:

>>> sudo kill -9 118238
kill: (118238): No such process

How can I get rid of this ghost process without interupting others?

Vitrescence

Answer 1

I have found the solution in this answer: https://mcmap.net/q/853183/-gpu-ram-occupied-but-no-pids.

First, I run sudo fuser -v /dev/nvidia* to see all processes are using my GPU RAM that nvidia-smi has failed to show.

Then, I saw some "ghost" Python processes. And after killing it, the GPU RAM was free up.

Vitrescence

To just kill all python processes: pkill -9 python3 – Nikaniki

that didn't work all the time... pkill .py kills all python processes a different way! – Nikaniki