Redis sentinel not connect to master: Error: READONLY You can't write against a read only slave?

Question

I have this redis replication with sentinel architect:

5 servers: .1, .2, .3, .4 and .5

• .1: nodejs app, redis master, redis sentinel
• .2: nodejs app, redis slave
• .3: nodejs app, redis slave
• .4: redis sentinel
• .5: redis sentinel

Sentinel handle failover as expected. redis-cli -h x.y.z.5 -p 26379 sentinel get-master-addr-by-name mymaster is always return .1 server.

My code to connect to this replication (I use ioredis):

let sentinels = [
  { host: process.env.REDIS_SENTINEL_1, port: process.env.REDIS_PORT },
  { host: process.env.REDIS_SENTINEL_2, port: process.env.REDIS_PORT },
  { host: process.env.REDIS_SENTINEL_3, port: process.env.REDIS_PORT }
]
store = new Redis({
  name: 'mymaster',
  sentinels: sentinels
})
store.on('ready', () => {
  store.set('foo', new Date())
  store.get('foo', function (err, result) {
    console.log('redis: get foo: ', result)
  });
});

Then, .1 nodejs app run ok, but in .2 and .3, the error occurs with this log:

 events.js:161
       throw er; // Unhandled 'error' event
       ^

 Error: READONLY You can't write against a read only slave.

P/s: my redis servers are binding in both server IP and 127.0.0.1. Because, without 127.0.0.1, I meet this error:

Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379

So, I guess that, my app isn't connect to redis server via sentinel, but it connect directly with localhost.

Any help are appriciate!

Update: log when I turn on the DEBUG=ioredis:*:

2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:redis status[x.y.z.5:26379]: connecting -> connect
2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:redis status[x.y.z.5:26379]: connect -> ready
2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:connection send 1 commands in offline queue
2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:redis write command[0] -> sentinel(get-master-addr-by-name,mymaster)
2|MyApp     | events.js:161
2|MyApp     |       throw er; // Unhandled 'error' event
2|MyApp     |       ^
2|MyApp     | Error: READONLY You can't write against a read only slave.
2|MyApp     |     at JavascriptReplyParser._parseResult (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:43:16)
2|MyApp     |     at JavascriptReplyParser.try_parsing (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:114:21)
2|MyApp     |     at JavascriptReplyParser.run (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:126:22)
2|MyApp     |     at JavascriptReplyParser.execute (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:107:10)
2|MyApp     |     at Socket.<anonymous> (/home/demo/demo_api/source/node_modules/redis/index.js:131:27)
2|MyApp     |     at emitOne (events.js:96:13)
2|MyApp     |     at Socket.emit (events.js:189:7)
2|MyApp     |     at readableAddChunk (_stream_readable.js:176:18)
2|MyApp     |     at Socket.Readable.push (_stream_readable.js:134:10)
2|MyApp     |     at TCP.onread (net.js:551:20)
2|MyApp     | [nodemon] app crashed - waiting for file changes before starting...
2|MyApp     | sentinel production
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[localhost:6379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[x.y.z.5:26379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,mymaster)
2|MyApp     | sentinel production
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[localhost:6379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[x.y.z.5:26379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,mymaster)

Update 2: one more thing, I run the app in my local machine with the production environment and everything work like charm!!!

Magritte

Answer 1

If the above error occurs, it means that the current Redis service is read-only and has no write permission. It is estimated that the service is used as a slave database.

Solution 1

Open the configuration file corresponding to the Redis service and modify the value of the attribute slave-read-only to no, so that it can be written.

Solution 2

Or a faster way is through run this command

redis-cli -h 127.0.0.1 -p 6379 slaveof no one

Command to stop the current Redis service from receiving synchronization from other Redis services, and at the same time upgrade itself to the main database.

Trimer

Answer 2

Yeah, finally found.

It's my mistake when an old code use the redis module and try to connect directly to localhost.

Replace it to ioredis instance and it work like charm :)

Magritte

Answer 3

By connecting to "localhost", .2 and .3 are talking to their local read-only save. You don't "connect via sentinel". You connect to sentinel, get the master IP, then connect to that IP and port. For some reason you are trying to connect to localhost, not the IP being reported by sentinel. As to why, I can't say as the information is not available in your question.

Sentinel returns an IP address, not "localhost", so you are not using the return from sentinel but instead are somewhere coded to use "localhost", which will produce the behavior you report. You'll need to find that and fix it.

Microsome

Thank for your comment, I think so too, however, in my code, I've tried to connect via sentinel-as I showed above. And I don't know why it try to connect to localhost??? – Magritte

Update 2: one more thing, I run the app in my local machine with the production environment and everything work like charm!!! – Magritte

In my case the issue was because of replicaof localhost 6380. When replaced that with replicaof 127.0.0.1 6380 the issue was solved. – Program