Redis – Failed opening .rdb for saving: Permission denied

Question

I am using redis version 3.0.6. The redis-server process is being run by the redis user.

Suddenly from 5 days after 24 hours redis began failing "opening .rdb for saving." It was working properly before this.

As you can see in the snippet from the logs below, Redis was behaving normally, and then started failing. Power-cycling the server later resolved the issue.

1427:M 24 May 01:09:05.102 * Background saving started by pid 2493    
2493:C 24 May 01:09:34.916 * DB saved on disk
2493:C 24 May 01:09:34.917 * RDB: 310 MB of memory used by copy-on-write
1427:M 24 May 01:09:34.950 * Background saving terminated with success
1427:M 24 May 01:14:35.026 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:14:35.036 * Background saving started by pid 2494
2494:C 24 May 01:15:04.329 * DB saved on disk
2494:C 24 May 01:15:04.330 * RDB: 298 MB of memory used by copy-on-write
1427:M 24 May 01:15:04.408 * Background saving terminated with success
1427:M 24 May 01:20:05.008 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:20:05.018 * Background saving started by pid 2499
2499:C 24 May 01:20:33.830 * DB saved on disk
2499:C 24 May 01:20:33.831 * RDB: 330 MB of memory used by copy-on-write
1427:M 24 May 01:20:33.843 * Background saving terminated with success
1427:M 24 May 01:23:46.966 # Failed opening .rdb for saving: Read-only file system
1427:M 24 May 01:25:34.029 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:25:34.038 * Background saving started by pid 2500
2500:C 24 May 01:25:34.038 # Failed opening .rdb for saving: Read-only file system
1427:M 24 May 01:25:34.139 # Background saving error
1427:M 24 May 01:25:40.059 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:25:40.064 * Background saving started by pid 2501
2501:C 24 May 01:25:40.064 # Failed opening .rdb for saving: Read-only file system
1427:M 24 May 01:25:40.165 # Background saving error
1427:M 24 May 01:25:46.080 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:25:46.085 * Background saving started by pid 2502
2502:C 24 May 01:25:46.085 # Failed opening .rdb for saving: Read-only file system
1427:M 24 May 01:25:46.186 # Background saving error
1427:M 24 May 01:25:52.100 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:25:52.105 * Background saving started by pid 2503
2503:C 24 May 01:25:52.105 # Failed opening .rdb for saving: Read-only file system
1427:M 24 May 01:25:52.206 # Background saving error

So, my question: how could this happen? Please give me proper solution for this.

Twentieth

Seems to be a duplicate of #22161253 – Urolith

@Urolith no, in that question redis failed to start at startup, here seems to be a different story – Any

@beena did you make sure only a single redis server was running? – Any

and where is your redis server hosted? – Any

and can you consistently reproduce the problem? without it any answer would be just a guess – Any

Problem comes rarely. – Twentieth

@BeenaShetty : check this link . hope it can be useful for your query. serverfault.com/questions/800295/… – Rebane

Answer 1

The "Read-only file system" I think is the key here. It's possible the device it's trying to write to is mounted incorrectly but since it happened randomly, the system may have forced the filesystem into readonly mode. There's a number of conditions that can trigger the operating system to put the filesystem into a read-only mode. This can mean that the filesystem became corrupt or there was some other filesystem consistency issue. If you're hosting on a cloud provider and the disk is network-backed like EBS in AWS, this can be triggered by a temporary network issue. Sometimes the issues are momentary and either force remounting the partition (or power cycling the server) will fix the issue. Other times it's permanent, but since your server came back up just fine, that would appear to not be the case. But the true fix for this would lie in your hardware setup which wasn't detailed.

This answer is related albeit thin on the "why": Failed opening the RDB file ... Read-only file system

Denaturalize

If it happens again, run dmesg to see the logs, you will probably find why the system set the filesystem in read only mode. As Ryan said, it can be a network issue if the partition is a network mount, or maybe the disk (or its controller) is faulty. – Blackmarket

Answer 2

After an upgrade.. (Ubuntu 14.04 LTS)

I had redis complain of this.. the file system was not RO. It was fine.

kill -9 REDIS-PROCESS # Otherwise it wouldn't die. looping on the error.

Deleted the dump.rdb file that already existed..

Started REDIS again, and the problem appeared to go away. (I only just did it.. so things may come back..)

It looks like it may have been an upgrade issue..

Michelemichelina

How did it go? I'm experiencing the same random RO error. – Maxilliped

Answer 3

1. you can check your redis.conf, in this configure file you can find where the dbfilename is,

2. give the permission 755 'dir' which include dbfilename, it is /var/lib/redis (centos), and the user and group to 'redis', but it should be 644 for the files in the dir.

3. restart redis.

Columniation

Answer 4

In my case I was having similar issue using redis-stack-server, the .rdb file was getting created in location /var/lib/redis-stack

Changing the permission as below solved the problem.

RUN chmod -R 777 /var/lib/redis-stack 

Yawn