Failed opening the RDB file ... Read-only file system

Question

I'm trying to perform a save or bgsave on my redis instance to run through the backup/restore process. I'm getting errors when I try to save however:

532:M 28 Jun 23:58:30.396 # Failed opening the RDB file backup.rdb (in server root dir /var/lib/redis) for saving: Read-only file system

Permissions on the /var/lib/redis folder:

$@/var/lib$ ls -artl | grep redis
drwxrwxrwx  3 redis    redis    4096 Jun 28 23:58 redis

Permissions on the /var/lib folder:

$@/var$ ls -artl | grep lib
drwxrwxrwx 31 root root  4096 Jun 28 23:44 lib

Permissions on the /var folder:

$@/$ ls -artl | grep var
drwxrwxrwx  11 root root  4096 Jul 18  2016 var

Redis CLI output for config get dir:

1) "dir"
2) "/var/lib/redis"

Redis CLI output for config get dbfilename:

1) "dbfilename"
2) "backup.rdb"

Error from redis:

532:M 28 Jun 23:58:30.396 # Failed opening the RDB file backup.rdb (in server root dir /var/lib/redis) for saving: Read-only file system

Any help would be much appreciated!

Sentimental

Also, in case anyone is wondering, the /var/lib/redis/ folder is empty. – Sentimental

Just to comment that this can also happen if there is not enough memory. In my case, Redis was complaining about a read-only filesystem for /var/spool/cron which was in no configuration file at all. Increasing the VM from 1GB RAM to 2GB and rebooting made it work. Not really a solution, but maybe that saves someone else. – Dialectology

Answer 1

You need to add the following to your /etc/systemd/system/redis-server unit file:

ReadWriteDirectories=-/var/lib/redis

Note that /var/lib/redis is the default, but if in your /etc/redis/redis.conf you set a different dir config option, you will need to set ReadWriteDirectories to that.

Browning

In Ubuntu 16.04 the unit file is located at /etc/systemd/system/redis.service – Psalmist

Great tip, thanks! I did this and everything worked fine for a while, but then suddenly that line simply disappeared from the /etc/systemd/system/redis.service file. Any idea how that could have happened? Is this file automatically regenerated/overwritten in some cases? – Derwon

I found out this issue, it turns out this solution is not entirely correct, please see this answer for the correct way to do this: serverfault.com/a/943335/498459 – Derwon

Remember to run sudo systemctl daemon-reload after the editing the file. – Hydantoin

Do not edit the default unit file. Instead create a drop-in with sytemctl edit. – Siana

My config file already has "ReadWriteDirectories=-/var/lib/redis", what to do? – Arrhythmia

Answer 2

Error says Read-only file system So, check the mounting (/ or /var) of file system, if it is read only, remount the FS with rw mode(read and write mode) Take backup of important data before mounting.

Arian

Thanks for the answer! Unfortunately, the mounting seems okay. Redis server is being run by the redis user, and when I sudo su - redis I can touch /var/lib/redis/blah.blah with no errors, and see the file get created. – Sentimental

I am always seeing this problem. I've verified by looking at /proc/mount that the directory's filesystem has the rw option. – Browning