Kafka Consumer startup error: Failed to add leader for partitions [calls,0] - NotLeaderForPartitionException

Question

Note: this is for an older version : We are running kafka_2.9.2-0.8.1

When attempting to run the kafka-console-consumer.bat against an existing topic on windows7 we receive "failed to add leader" and "NotLeaderForPartition" exceptions

Here is the command line

set GROUP=group1234
kafka-console-consumer.bat --group %GROUP%  --zookeeper localhost:2181 --topic calls --from-beginning

Here are the errors:

[2014-05-26 15:02:12,997] WARN [group1234_S80035683-SC01-1401141732400-98745e28-leader-finder-thread], Failed to add leader for partitions [calls,0];
will retry (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread:89)
kafka.common.NotLeaderForPartitionException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at java.lang.Class.newInstance(Class.java:374)
        at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:73)
        at kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:160)
        at kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
        at kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
        at kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
        at scala.collection.immutable.Map$Map1.foreach(Map.scala:119)
        at kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
        at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
        at kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
        at scala.collection.immutable.Map$Map1.foreach(Map.scala:119)
        at kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)

And we are unable to consume any messages in the end.

We are running kafka_2.9.2-0.8.1

Note: Zookeeper is seeing the Consumer attempt to connect: I go into zkCli and can see the new group1234

[zk: localhost:2181(CONNECTED) 2] ls2 /consumers/group1234
[owners, ids]
cZxid = 0x123
ctime = Mon May 26 15:02:12 PDT 2014
mZxid = 0x123
mtime = Mon May 26 15:02:12 PDT 2014
pZxid = 0x128
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 2

And here is info on the requested calls topic in ZK:

[zk: localhost:2181(CONNECTED) 7] ls2 /brokers/topics/calls
[partitions]
cZxid = 0x18
ctime = Sat May 24 23:15:16 PDT 2014
mZxid = 0x18
mtime = Sat May 24 23:15:16 PDT 2014
pZxid = 0x1c
cversion = 1
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 36
numChildren = 1

In case there were corruption on the topic, I just dropped it in ZK and then recreated it via kafka-topics.bat. Here is the new ZK output

[zk: localhost:2181(CONNECTED) 15] ls2 /brokers/topics/calls
[]
cZxid = 0x136
ctime = Mon May 26 16:02:51 PDT 2014
mZxid = 0x136
mtime = Mon May 26 16:02:51 PDT 2014
pZxid = 0x136
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 36
numChildren = 0

Brassie

Any luck in solving this? – Tommyetommyrot

There were other bugs on Kafka - some not just on Windows. E.g. no ability to delete a topic. I re-coded in RabbitMQ. Eventually we will go back to Kafka (when these bugs fixed). – Brassie

Voted to close as this question has not seen activity since 2014, and we are now many Kafka versions further. – Bloodletting

@DennisJaheruddin I will agree with closing this for the reason you stated. Kafka v 0.8.X had so many serious bugs. Oh wait ! There is NO appropriate reason available for closing: it is certainly NOT off topic .. just too dated. Soo.. I'm retracting my agreement temporarily. Can you check up on Meta.stackoverflow to see what is policy simply on questions that have to do with OLD VERSIONS? I mean this happens regularly .. are OTHER questions closed for that reason? I honestly do not know offhand. – Brassie

@javadba I am not happy with the naming of off-topic. But within this there is a subcategory for "Questions about a problem that can no longer be reproduced". I think that is a reasonable fit unless people are still running into it. – Bloodletting

As a compromise I will make the commentary about the kafka version stand -out – Brassie

Thanks for the update, but as we are now again 2 years further and the version really shouldn't be used anymore, I wonder if the time has come to close it? Unless anyone thinks this is still relevant for future visitors. – Bloodletting

@DennisJaheruddin I certainly do not agree to close for Off-Topic - or for any other reason that indicates a poor fit for SOF. If you care enough then please ask on Meta what the preferred approach is to annotate that it is old version. I think the mention of "old version" is the most common and appropriate way to do it : so nothing need be done – Brassie

Answer 1

A search shows that now, 7 years later, this is no longer a known problem for current versions.

There have also been multiple patches which resolve errors that may or may not be the same, and it is almost certain that one of these fixed the issue.

As such the only practical solution for anyone seems to be upgrading to a newer version. (For Kafka, Zookeeper as well as Windows.)

If the problem persists in currently relevant versions please ask a new question as the root cause is unlikely to be the same.

Bloodletting

Hi Dennis - this is a fair way to handle the aspect of the question referring to an old version. I will award – Brassie