Fails to `mkdir /mnt/vzsnap0` for Container Backups with Permission Denied

Question

This is all done as the root user.

The script for backups at /usr/share/perl5/PVE/VZDump/LXC.pm sets a default mount point

my $default_mount_point = "/mnt/vzsnap0";

But regardless of whether I use the GUI or the command line I get the following error:

ERROR: Backup of VM 103 failed - mkdir /mnt/vzsnap0: 
Permission denied at /usr/share/perl5/PVE/VZDump/LXC.pm line 161.

And lines 160 - 161 in that script is:

my $rootdir = $default_mount_point;
mkpath $rootdir;

After the installation before I created any images or did any backups I setup two things.

• (1) SSHFS mount for /mnt/backups
• (2) Added all other drives as Linux LVM

What I did for the drive addition is as simple as:

pvcreate /dev/sdb1
pvcreate /dev/sdc1
pvcreate /dev/sdd1
pvcreate /dev/sde1
vgextend pve /dev/sdb1
vgextend pve /dev/sdc1
vgextend pve /dev/sdd1
vgextend pve /dev/sde1
lvextend pve/data /dev/sdb1
lvextend pve/data /dev/sdc1
lvextend pve/data /dev/sdd1
lvextend pve/data /dev/sde1

For the SSHFS instructions see my blog post on it: https://6ftdan.com/allyourdev/2018/02/04/proxmox-a-vm-server-for-your-home/

---

Here are filesystem directory permission related files and details.

cat /etc/fstab

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

df -h

Filesystem                                Size  Used Avail Use% Mounted on
udev                                      7.8G     0  7.8G   0% /dev
tmpfs                                     1.6G  9.0M  1.6G   1% /run
/dev/mapper/pve-root                       37G  8.0G   27G  24% /
tmpfs                                     7.9G   43M  7.8G   1% /dev/shm
tmpfs                                     5.0M     0  5.0M   0% /run/lock
tmpfs                                     7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/fuse                                  30M   20K   30M   1% /etc/pve
[email protected]:/mnt/raid/proxmox_backup  1.4T  725G  672G  52% /mnt/backups
tmpfs                                     1.6G     0  1.6G   0% /run/user/0

ls -dla /mnt

drwxr-xr-x 3 root root 0 Aug 12 20:10 /mnt

ls /mnt

backups

ls -dla /mnt/backups

drwxr-xr-x 1 1001 1002 80 Aug 12 20:40 /mnt/backups

---

The command that I desire to succeed is:

vzdump 103 --compress lzo --node ProxMox --storage backup --remove 0 --mode snapshot

For the record the container image is only 8GB in size.

Cloning containers does work and snapshots work.

---

Q & A

---

Q) How are you running the perl script?

A) Through the GUI you click on Backup now, then select your storage (I have backups and local and the both produce this error), then select the state of the container (Snapshot, Suspend, Stop each produce the same error), then compression type (none, LZO, and gzip each produce the same error). Once all that is set you click Backup and get the following output.

INFO: starting new backup job: vzdump 103 --node ProxMox --mode snapshot --compress lzo --storage backups --remove 0
INFO: Starting Backup of VM 103 (lxc)
INFO: Backup started at 2019-08-18 16:21:11
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: Passport
ERROR: Backup of VM 103 failed - mkdir /mnt/vzsnap0: Permission denied at /usr/share/perl5/PVE/VZDump/LXC.pm line 161.
INFO: Failed at 2019-08-18 16:21:11
INFO: Backup job finished with errors
TASK ERROR: job errors

From this you can see that the command is vzdump 103 --node ProxMox --mode snapshot --compress lzo --storage backups --remove 0 . I've also tried logging in with a SSH shell and running this command and get the same error.

---

Q) It could be that the directory's "immutable" attribute is set. Try lsattr / and see if /mnt has the lower-case "i" attribute set to it.

A) root@ProxMox:~# lsattr /

--------------e---- /tmp
--------------e---- /opt
--------------e---- /boot
lsattr: Inappropriate ioctl for device While reading flags on /sys
--------------e---- /lost+found
lsattr: Operation not supported While reading flags on /sbin
--------------e---- /media
--------------e---- /etc
--------------e---- /srv
--------------e---- /usr
lsattr: Operation not supported While reading flags on /libx32
lsattr: Operation not supported While reading flags on /bin
lsattr: Operation not supported While reading flags on /lib
lsattr: Inappropriate ioctl for device While reading flags on /proc
--------------e---- /root
--------------e---- /var
--------------e---- /home
lsattr: Inappropriate ioctl for device While reading flags on /dev
lsattr: Inappropriate ioctl for device While reading flags on /mnt
lsattr: Operation not supported While reading flags on /lib32
lsattr: Operation not supported While reading flags on /lib64
lsattr: Inappropriate ioctl for device While reading flags on /run

---

Q) Can you manually created /mnt/vzsnap0 without any issues?

A) root@ProxMox:~# mkdir /mnt/vzsnap0

mkdir: cannot create directory ‘/mnt/vzsnap0’: Permission denied

---

Q) Can you replicate it in a clean VM ?

A) I don't know. I don't have an extra system to try it on and I need the container's I have on it. Trying it within a VM in ProxMox… I'm not sure. I suppose I could try but I'd really rather not have to just yet. Maybe if all else fails.

---

Q) If you look at drwxr-xr-x 1 1001 1002 80 Aug 12 20:40 /mnt/backups, it looks like there are is a user with id 1001 which has access to the backups, so not even root will be able to write. You need to check why it is 1001 and which group is represented by 1002. Then you can add your root as well as the user under which the GUI runs to the group with id 1002.

A) I have no problem writing to the /mnt/backups directory. Just now did a cd /mnt/backups; mkdir test and that was successful.

Gebhardt

I have a theory that if I had done some initial work on the system such as installing and building a container before adding the remote mounted storage that perhaps it would have done some "first time setup" tasks. – Gebhardt

How are you running the perl script? – Ladylike

@TarunLalwani I've updated the details to answer your question. – Gebhardt

If you look at drwxr-xr-x 1 1001 1002 80 Aug 12 20:40 /mnt/backups, it looks like there are is a user with id 1001 which has access to the backups, so not even root will be able to write. You need to check why it is 1001 and which group is represented by 1002. Then you can add your root as well as the user under which the GUI runs to the group with id 1002. This thread would help cyberciti.biz/faq/unix-linux-id-command-examples-usage-syntax/… – Ladylike

Can you manually created /mnt/vzsnap0 without any issues? – Gateshead

Can you replicate it in a clean VM ? – Appel

@LucasRamage No, I'm unable to. – Gebhardt

@TarunLalwani … that doesn't seem to be the case. – Gebhardt

@EdwardAung … perhaps, not for now though. – Gebhardt

Can you mount to a different directory than /mnt? Can you create any other directory in /mnt like e.g. mkdir /mnt/wasNeverThere ? – Grecian

have you tried to launch the gui as root? – Bouffard

Can you try to see if there is an "acl" in your filesystem: "getfacl /mnt". Can you use debugfs to see if you can create a directory through it: "debugfs /dev/mapper/pve-root" and then "mkdir /mnt/test"? – Sheba

Answer 1

From the message

mkdir /mnt/vzsnap0: Permission denied

it is obvious the problem is the permissions for /mnt directory.

It could be that the directory `s "immutable" attribute is set.

Try lsattr / and see if /mnt has the lower-case "i" attribute set to it.

As a reference:

The lower-case i in lsattr output indicates that the file or directory is set as immutable: even root must clear this attribute first before making any changes to it. With root access, you should be able to remove this with chattr -i /mnt, but there is probably a reason why this was done in the first place; you should find out what the reason was and whether or not it's still applicable before removing it. There may be security implications.

So, if this is the case, try:

chattr -i /mnt

to remove it.

References

lsattr output

According to inode flags—attributes manual page:

FS_IMMUTABLE_FL 'i': The file is immutable: no changes are permitted to the file contents or metadata (permissions, timestamps, ownership, link count and so on). (This restriction applies even to the supe‐ ruser.) Only a privileged process (CAP_LINUX_IMMUTABLE) can set or clear this attribute.

Sheba

Thanks for giving another direction in which to look. It seems the problem runs deeper than this as lsattr cannot read from /mnt – Gebhardt

Answer 2

As long as the bounty is still up I'll give it to a legitimate answer that fixes the problem described here.

What I'm writing here for you all is a work around I've thought of which works. Note, it is very slow.

Since I am able to write to the /mnt/backups directory, which exists on another system on the network, I went ahead and changed the Perl script to point to /mnt/backups/vzsnap0 instead of /mnt/vzsnap0.

Bounty remains for anyone who can get the /mnt directory to work for the mount path to successfully mount vzsnap0 for the backup script..

Gebhardt

Answer 3

1) Perhaps your "/mnt/vzsnap0" is mounted as read only?

It may tell from your:

/dev/pve/root / ext4 errors=remount-ro 0 1 'errors=remount-ro' means in case of mistake remounting the partition like readonly. Perhaps this setting applies for your mounted filesystem as well.

Can you try remounting the drive as in the following link? https://askubuntu.com/questions/175739/how-do-i-remount-a-filesystem-as-read-write

And if that succeeds, manually create the directory afterwards?

2) If that didn't help: https://www.linuxquestions.org/questions/linux-security-4/mkdir-throws-permission-denied-error-in-a-directoy-even-with-root-ownership-and-777-permission-4175424944/ There, someone remarked:

What is the filesystem for the partition that contains the directory.[?] Double check the permissions of the directory, or whether it's a symbolic link to another directory. If the directory is an NFS mount, rootsquash can prevent writing by root.

Check for attributes (lsattr). Check for ACLs (getfacl). Check for selinux restrictions. (ls -Z)

If the filesystem is corrupt, it might be initially mounted RW but when you try to write to a bad area, change to RO.

Jemine

Answer 4

Great, turns out this is a pretty long-standing issue with Ubuntu Make which is faced by many people.

I saw a workaround mentioned by an Ubuntu Developer in the above link.

Just follow the below steps:

sudo -s
unset SUDO_UID
unset SUDO_GID

Then run umake to install your application as normal.

you should now be able to install to any directory you want. Works flawlessly for me.

Beshore

Answer 5

try ls laZ /mnt to review the security context, in case SE Linux is enabled. relabeling might be required then. errors=remount-ro should also be investigated (however, it is rather unlikely lsattr would fail, unless the /mnt inode itself is corrupted). Creating a new directory inode for these mount-points might be worth a try; if it works, one can swap them.

Anticlimax

Answer 6

Just change /mnt/backups to /mnt/sshfs/backups

And the vzdump will work.

Reeher