Zookeeper refuses Kafka connection from an old client

Question

I have a cluster configuration using Kubernetes on GCE, I have a pod for zookeeper and other for Kafka; it was working normally until Zookeeper get crashed and restarted, and it start refusing connections from the kafka pod:

Refusing session request for client /10.4.4.58:52260 as it has seen zxid 0x1962630

The complete refusal log is here:

2017-08-21 20:05:32,013 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /10.4.4.58:52260
2017-08-21 20:05:32,013 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@882] - Connection request from old client /10.4.4.58:52260; will be dropped if server is in r-o mode
2017-08-21 20:05:32,013 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@901] - Refusing session request for client /10.4.4.58:52260 as it has seen zxid 0x1962630 our last zxid is 0xab client must try another server
2017-08-21 20:05:32,013 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /10.4.4.58:52260 (no session established for client)

Desmarais

Answer 1

Because the kafka maintain a zookeeper session which remember the last zxid it has seen. So when the zookeeper sevice go down and come again, the zk's zxid begin from a smaller value. and ZKserver think the kafka has seen a bigger zxid, so it refuse it.

Have a try to restart the kafka.

Bermudez

Answer 2

For the record, I had this problem and all my kafka were off.
But, my kafka-manager was still up and listening on zookeepers. Turning it off resolved the issue.

Alcazar

How did you turn off the manager? turning off the kafka server doesn't seem to working – Doting

Answer 3

Related to the answer from @GuangshengZuo.... Steps

1. Stop any residual zookeeper instances - zookeeper-server-stop.bat
2. Start a fresh zookeeper- zookeeper-server-start.bat .\config\zookeeper.properties This will do

Incomprehensible

Answer 4

I had this problem. This is how I solved it. first I shut down both zookeeper and kafka then I started zookeeper, then I started kafka.

This worked on Centos 7. It assumes you are one level above the bin directory that the kafka scripts are in. I used 2 windows, one for the zookeeper process and one for the kafka process. This was an experiment to see what would work and what wouldnt. You maybe want to have things run in the background and redirect the output to a log file for use in production etc....

the exact steps:

 1. shut down kafka and zookeeper
    ./bin/kafka-server-stop.sh  ./bin/zookeeper-server-stop.sh 
    
 2. start zookeeper
    ./bin/zookeeper-server-start.sh ./config/zookeeper.properties
    
 3. start kafka 
    ./bin/kafka-server-start.sh ./config/server.properties

Gymnastics