XFS No space left on device

Question

I have a server setup of an XFS partition on LVM. While copying files to the home partition, "No space left on device" is displayed.

df -h displays sufficient space:

/dev/mapper/prod--vg-home     35G   21G   15G  60% /home

df -i also displays sufficient inodes:

/dev/mapper/prod--vg-home   36700160  379390 36320770    2% /home

I did verify the impact of changing the maximum percentage of inodes:

xfs_growfs -m 25 /dev/mapper/prod--vg-home

This amount can easily be decreased and increased.

While experimenting with this setting, I noticed that decreasing it to 3% and increasing it again to 25%, and deleting some files, allows me to add a lot more files again.

xfs_info displays:

meta-data=/dev/mapper/prod--vg-home isize=256    agcount=14, agsize=655360 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=9175040, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

I did read about 64-bit inodes, but it seems to be applicable only for large drives (over 1TB).

Is there any other setting which could cause the "No space left on device" message.

Thank you

Fullerton

Answer 1

There is a bug with xfs_growfs which causes inodes to not be properly distributed across a partition. The solution is to simply remount with the inode64 option. For example, if this was the /dev/vda1, you would do the following:

mount -o remount,inode64 /dev/vda1

You can find more information about the bug at the following link:

http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_receive_No_space_left_on_device_after_xfs_growfs.3F

Heft

Answer 2

You should also check possible "deleted" files and restart some processes: lsof -nP +L1

Kellikellia

This happens all the time for me on Foreman Smartproxy: [root@smartproxy ~]# lsof -nP +L1 | grep deleted | wc -l 41 – Rainmaker