Kafka consumer gets stuck after exceeding max.poll.interval.ms

Question

When the consumer does not receives a message for 5 mins (default value of max.poll.interval.ms 300000ms) the consumer comes to a halt without exiting the program. The consumer process hangs and does not consume any more messages.

The following error message gets logged

MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 255ms (adjust max.poll.interval.ms for long-running message processing): leaving group

I see that ErrMaxPollExceeded is defined here in confluent-kafka-go but unable to find where it is getting raised.

If any such error is raised, why does the program not exit ?

• Kafka v1.1.0
• librdkafka v1.0.0
• confluent-kafka-go (master)

Config used for kafka.Consumer

{
    "bootstrap.servers":    "private.kafka.host",
    "group.id":             "foo.bar",
    "auto.offset.reset":    "earliest",
    "enable.auto.commit":   false,
}

Tannertannery

Are you calling Poll() or ReadMessage() with a timeout longer than the max.poll.interval.ms? If so, you're probably hitting an issue in librdkafka which is fixed in librdkafka v1.1.0. – Unreserved

@Unreserved No, I'm calling Poll() with a timeout of 100 ms – Tannertannery

Hi @Tannertannery did you find solution, please let me know am facing same the issue. – Rata

Answer 1

It looks like this is a reported issue to the confluent-kafka-go client: https://github.com/confluentinc/confluent-kafka-go/issues/344

It is marked as resolved by this change which should have been released in librdkafka version 1.1.0: https://github.com/edenhill/librdkafka/commit/80e9b1ee0c0e477c02f18b130c4302240711a88d

It looks like that change should also have been released in confluent-kafka-go version 1.1.0: https://github.com/confluentinc/confluent-kafka-go/tree/v1.1.0

Some comments in the issue still report that this is an issue experiences on versions >1.1.0.

A commenter suggests a work around:

Using v1.5.2. Also calling ReadMessage(-1) in an infinite loop, and not seeing rejoining after consumer leaving group, worked around it by setting timeout to be less than max.poll.interval.ms instead of -1, but wondering why it's not rejoining as expected.

https://github.com/confluentinc/confluent-kafka-go/issues/344#issuecomment-745014864

Scharff