Unable to gossip with any seeds but continuing since node is in its own seed list

Question

To remove a node from 2 node cluster in AWS I ran

nodetool removenode <Host ID>

After this I was supposed to get my cluster back if I put all the cassandra.yaml and cassandra-rackdc.properties correctly. I did it but still, I am not able to get back my cluster.

nodetool status is displaying only one node.

significant system.log on cassandra is :

INFO  [main] 2017-08-14 13:03:46,409 StorageService.java:553 - Cassandra version: 3.9
INFO  [main] 2017-08-14 13:03:46,409 StorageService.java:554 - Thrift API version: 20.1.0
INFO  [main] 2017-08-14 13:03:46,409 StorageService.java:555 - CQL supported versions: 3.4.2 (default: 3.4.2)
INFO  [main] 2017-08-14 13:03:46,445 IndexSummaryManager.java:85 - Initializing index summary manager with a memory pool size of 198 MB and a resize interval of 60 minutes
INFO  [main] 2017-08-14 13:03:46,459 MessagingService.java:570 - Starting Messaging Service on /172.15.81.249:7000 (eth0)
INFO  [ScheduledTasks:1] 2017-08-14 13:03:48,424 TokenMetadata.java:448 - Updating topology for all endpoints that have changed
WARN  [main] 2017-08-14 13:04:17,497 Gossiper.java:1388 - Unable to gossip with any seeds but continuing since node is in its own seed list
INFO  [main] 2017-08-14 13:04:17,499 StorageService.java:687 - Loading persisted ring state
INFO  [main] 2017-08-14 13:04:17,500 StorageService.java:796 - Starting up server gossip

Content of files:

cassandra.yaml : https://pastebin.com/A3BVUUUr

cassandra-rackdc.properties: https://pastebin.com/xmmvwksZ

system.log : https://pastebin.com/2KA60Sve

netstat -atun https://pastebin.com/Dsd17i0G

Both the nodes have same error log.

All required ports are open.

Any suggestion ?

Cristophercristy

how many nodes were there in cluster earlier... did you change seed node (if seed node was removed) – Beelzebub

there were two nodes before and after nodes were removed I did not change seeds. both nodes have seeds like node 1 had - seeds: "node1,node2" node 2 had - seeds: "node2,node1"` – Cristophercristy

Have started both cluster node? if not than do it first and check. – Tomkin

I did it already but still same problem . – Cristophercristy

You have still one node running? Could you post the logs from cassandas system.log on the running and the starting node when you try to readd your node? – Fabrienne

Answer 1

It's usually a best practice to have one seed node per DC if you have just two nodes available in your datacenter. You shouldn't make every node a seed node in this case.

I noticed that node1 has - seeds: "node1,node2" and node2 has - seeds: "node2,node1" in your configuration. A node will start by default without contacting any other seeds if it can find it's IP address as first element in - seeds: ... section in the cassandra.yml configuration file. That's what you can also find in your logs:

... Unable to gossip with any seeds but continuing since node is in its own seed list ...

I suspect, that in your case node1 and node2 are starting without contacting each other, since they identify themselves as seed nodes.

Try to use just node1 for seed node in both instance's configuration and reboot your cluster. In case of node1 being down and node2 is up, you have to change - seeds: ... section in node1 configuration to point just to node2's IP address and just boot node1.

If your nodes can't find each other because of firewall misconfiguration, it's usually a good approach to verify if a specific port is accessible from another location. E.g. you can use nc for checking if a certain port is open:

nc -vz node1 7000

References and Links

See the list of ports Cassandra is using under the following link http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureFireWall.html

See also a detailed documentation on running multiple nodes with plenty of sample commands: http://docs.datastax.com/en/cassandra/2.1/cassandra/initialize/initializeMultipleDS.html

Rotor

Not working. Same problem. Even I followed this #23874660 – Cristophercristy

Did you check if the ports are open? Also what helped me is to execute netstat -nlp and check if the specific node is using the proper ethernet interface for accepting gossip messages. – Rotor

All ports are open and I ran netstat -atun so many times to check it. – Cristophercristy

I'll recheck your configurations again. Could you please also run ifconfig -a so we can see the configured interfaces in the nodes? Thanks! – Rotor

Glad that this is finally solved. Just FYI nc -vz node2 7000 would have done the same check as telnet Node2 7000 – Rotor

Answer 2

This is for future reference. My problem has been solved just by opening 7000 port for same security group in AWS. Although it was open but security group was something different. When I ran:

ec2-user@ip-Node1 ~]$ telnet Node2 7000
Trying Node2...
telnet: connect to address Node2: Connection timed out

I came to know the problem could be of the security group. And that is how it has been solved.

About seeds I am using IP of both the nodes, like this:

-seeds: "node1,node2"

It is same on both the nodes.

Cristophercristy