Redis sentinel doesn't auto-discover other sentinels

Question

I have redis configured as master with two slaves, each on a separate box. I also have a sentinel process running on each box. It's the setup as described in the documentation here:

http://redis.io/topics/sentinel#example-2-basic-setup-with-three-boxes

Each of the sentinels can connect to my master, and can see the slaves. They are able to independently detect if the master or slaves go down. The problem is that the sentinels can't detect each other.

I have verified that each sentinel is publishing a message to the __sentinel__:hello channel as expected, but it seems that none of them are actually receiving the messages from the other ones.

How do I get the sentinels to see each other?

Yokoyama

Answer 1

In configuring the sentinels, I configured sentinel once and then copied that configuration file onto all the others.

Before copying the config I must have started up sentinel and it re-wrote the config file with an id like:

sentinel myid c766a0df56e72eda48cea984

When I copied the file to the other sentinels it caused each of them to have the same id, which in turn caused them to ignore any messages from the auto-discovery with that same id. Removing the myid line from each config file and restarting each sentinel process fixed the issue.

Yokoyama

get same issue, fixed after removed myid – Arianism

For some reason I believed all sentinels should have the same id. I had the same issue and it worked after removing this line! :) – Anemia

Answer 2

Having the same myid in all the redis-sentinel.conf files is an issue, but i had the issue even though my ids were different (runid showing "" in the sentinel master mymaster results)

My issue stemmed from the fact that my sentinel way bound to both localhost (127.0.0.1) and the private (or public) IP

I believe that the discovered IP is then 127.0.0.1 (as it was the first one in the config) so for the other sentinels receiving the message, that meant that the other sentinels were the same as itself (even though it had a different myid)

The docs says:

Before adding a new sentinel to a master a Sentinel always checks if there is already a sentinel with the same runid or the same address (ip and port pair). In that case all the matching sentinels are removed, and the new added.

setting sentinel announce-ip <ip> or removing the bind command to 127.0.0.1 fixed the problem.

Disk

In our case, we had to also disable protected mode. – Rotl