Linux: Which process is causing "device busy" when doing umount? [closed]

Question

Linux: Which process is causing "device busy" when doing umount?

Stickybeak

You will need to provide some more information. There could be a number of different answers.... – Quanta

what info do you need? what are the possible answers? – Stickybeak

Reverting most recent edits. First off, shell scripting is as much programming as .net, SQL queries or iphone development. Secondly, removing the linux designation makes the question ambitious. BSD, OSX, etc. have slightly different semantics, and Solaris, etc. even more so. – Moonseed

don't forget NFS! You need to stop all NFS export or mount points on the drive you want to umount. NFS does NOT show up on lsof or fuser... so it's very tricky! – Crump

Wouldn't this be on-topic at Super User or Unix & Linux? – Appraise

Answer 1

Look at the lsof command (list open files) -- it can tell you which processes are holding what open. Sometimes it's tricky but often something as simple as sudo lsof | grep (your device name here) could do it for you.

Moonseed

In my case, grepping the device name didn't show any result. However, grepping the path into which the device was actually mounted did; so finally, by killing that process I was able to umount the device successfully. – Evasive

In my case the command didn't show anything. Exiting SSH client and logging in again did it. – Gebelein

You were probably inside that direction while trying to unmount it. – Gesso

lsof /mountpoint instead of lsof /mountpoint/ – Fabiano

There's an advantage to using lsof +f /dev/device rather than the mountpoint. See the same for kernel processes with anonymous inodes which may not show up even in that listing. – Burgess

Answer 2

Just in case... sometimes happens that you are calling umount from the terminal, and your current directory belongs to the mounted filesystem.

Urrutia

Got me too. You can recognise this if lsof lists the "bash" process (debian) – Minuet

fuser catches current working directories like this I believe. – Crump

it happens in mount point which sub dirctory has been bind mount. such as /data mount in /dev/vdb1 and /data/kubernetes/kubelet use by kubelet, which has some bind mount tmpfs /data/kubernetes/kubelet/pods/3959731f-4e0e-4d93-8930-f475cf6f877d/volumes/kubernetes.io~secret/default-token-x6957 – Greywacke

Answer 3

You should use the fuser command.

Eg. fuser /dev/cdrom will return the pid(s) of the process using /dev/cdrom.

If you are trying to unmount, you can kill theses process using the -k switch (see man fuser).

Zebe

FYI fuser will sometimes show things that lsof does not and vice versus. so you can not do just one or the other! – Crump

Didn't work, but you gave me the idea to kill my media player, and that worked, so thanks for the idea ;-) – Haversack

Answer 4

Check for open loop devices mapped to a file on the filesystem with "losetup -a". They wont show up with either lsof or fuser.

Handsome

Exactly! I had loop devices mounted onto .iso files on the partition in question. The umount -l /data actually worked - the FS disappeared - but later unmounting /dev/loop0 and /dev/loop1 hanged for a while. But now is seems all right. This answer is very important, because it helps people, who've known lsof and fuser for ages... – Overboard

This was key for me. in my case the docker devicemapper storage drive created /dev/loop0 /dev/loop1 loop devices. The only way I could get rid of them was listing devices using the dmsetup ls then finding the suspect docker devicemapper device docker-253:0-4291588-pool. dmsetup remove docker-253:0-4291588-pool got rid of it cleanly – Bramante

Answer 5

Also check /etc/exports. If you are exporting paths within the mountpoint via NFS, it will give this error when trying to unmount and nothing will show up in fuser or lsof.

Loewe

In our case it was our problem. – Ben

Why wouldn't it show up? Is this because nfs, is partly in the kernel ?(in contrast to being completely userspace) – Epigrammatist

Answer 6

lsof +f -- /mountpoint

(as lists the processes using files on the mount mounted at /mountpoint. Particularly useful for finding which process(es) are using a mounted USB stick or CD/DVD.

Sandstone

This helped me find a process whose current working directory prevented the AUTOCLEAR of a loop device mount. It wasn't listed in fuser or standard lsof. – Burgess

Even better is to use /dev/<device> rather than /mountpoint as /mountpoint will disappear after an umount -l. – Burgess

Answer 7

lsof and fuser are indeed two ways to find the process that keeps a certain file open. If you just want umount to succeed, you should investigate its -f and -l options.

Suellen

I had this problem recently and neither fuser or lsof would show anything using the device, but umount -l allowed me to unmount it. At least, it appears to (-l means Lazy Unmount, Detach the filesystem from the filesystem hierarchy now, and cleanup all references to the filesystem as soon as it is not busy anymore.) – Lewls

Answer 8

Open files

Processes with open files are the usual culprits. Display them:

lsof +f -- <mountpoint or device>

There is an advantage to using /dev/<device> rather than /mountpoint: a mountpoint will disappear after an umount -l, or it may be hidden by an overlaid mount.

fuser can also be used, but to my mind lsof has a more useful output. However fuser is useful when it comes to killing the processes causing your dramas so you can get on with your life.

List files on <mountpoint> (see caveat above):

fuser -vmM <mountpoint>

Interactively kill only processes with files open for writing:

fuser -vmMkiw <mountpoint>

After remounting read-only (mount -o remount,ro <mountpoint>), it is safe(r) to kill all remaining processes:

fuser -vmMk <mountpoint>

Mountpoints

The culprit can be the kernel itself. Another filesystem mounted on the filesystem you are trying to umount will cause grief. Check with:

mount | grep <mountpoint>/

For loopback mounts, also check the output of:

losetup -la

Anonymous inodes (Linux)

Anonymous inodes can be created by:

• Temporary files (open with O_TMPFILE)
• inotify watches
• [eventfd]
• [eventpoll]
• [timerfd]

These are the most elusive type of pokemon, and appear in lsof's TYPE column as a_inode (which is undocumented in the lsof man page).

They won't appear in lsof +f -- /dev/<device>, so you'll need to:

lsof | grep a_inode

For killing processes holding anonymous inodes, see: List current inotify watches (pathname, PID).

Burgess

Answer 9

That's exactly why the "fuser -m /mount/point" exists.

BTW, I don't think "fuser" or "lsof" will indicate when a resource is held by kernel module, although I don't usually have that issue..

Dapper

but this is exactly the issue I seem to have. how does one debug this? – Lot

Answer 10

If you still can not unmount or remount your device after stopping all services and processes with open files, then there may be a swap file or swap partition keeping your device busy. This will not show up with fuser or lsof. Turn off swapping with:

sudo swapoff -a

You could check beforehand and show a summary of any swap partitions or swap files with:

swapon -s

or:

cat /proc/swaps

As an alternative to using the command sudo swapoff -a, you might also be able to disable the swap by stopping a service or systemd unit. For example:

sudo systemctl stop dphys-swapfile

or:

sudo systemctl stop var-swap.swap

In my case, turning off swap was necessary, in addition to stopping any services and processes with files open for writing, so that I could remount my root partition as read only in order to run fsck on my root partition without rebooting. This was necessary on a Raspberry Pi running Raspbian Jessie.

Sot

Answer 11

lsof and fuser didn't give me anything either.

After a process of renaming all possible directories to .old and rebooting the system every time after I made changes I found one particular directory (relating to postfix) that was responsible.

It turned out that I had once made a symlink from /var/spool/postfix to /disk2/pers/mail/postfix/varspool in order to minimise disk writes on an SDCARD-based root filesystem (Sheeva Plug).

With this symlink, even after stopping the postfix and dovecot services (both ps aux as well as netstat -tuanp didn't show anything related) I was not able to unmount /disk2/pers.

When I removed the symlink and updated the postfix and dovecot config files to point directly to the new dirs on /disk2/pers/ I was able to successfully stop the services and unmount the directory.

Next time I will look more closely at the output of:

ls -lR /var | grep ^l | grep disk2

The above command will recursively list all symbolic links in a directory tree (here starting at /var) and filter out those names that point to a specific target mount point (here disk2).

Drippy

I am actually looking for symbolic links (the ^l in my command), your command might be faster but does not have the same effect. – Drippy

D'oh... that's because I typoed the command. It should be find /var -lname *disk2* – Mathis

Answer 12

Filesystems mounted on the filesystem you're trying to unmount can cause the target is busy error in addition to any files that are in use. (For example when you mount -o bind /dev /mnt/yourmount/dev in order to use chroot there.)

To find which file systems are mounted on the filesystem run the following:

mount | grep '/mnt/yourmount'

To find which files are in use the advice already suggested by others here:

lsof | grep '/mnt/yourmount'

Glazing