Kafka long polling

Question

I'm looking at kafka to implement a low-latency message queue, and I've been reading up about consumer long polling. However, there are no examples about how to actually use long polling, or what options you need to set to enable it. How do you enable long polling using the kafka java api?

Annalist

Answer 1

You don't have to enable it per say since it is the default behavior of the kafka consumer. What you need to set in your configuration is fetch.wait.max.ms.

Two values are important to achieve what you want:

• fetch.min.bytes: The broker will wait for this amount of data to fill BEFORE it sends the response to the consumer client.
• fetch.max.wait.ms: The broker will wait for this amount of time BEFORE sending a response to the consumer client, unless it has enough data to fill the response (fetch.message.min.bytes)

A response will be sent to the consumer once either of these values is reached. The longest possible fetch request time will always be fetch.max.wait.ms.

Look for more configuration options in Kafka Consumer Configs

The kafka user mailing list is a also good option for questions like this.

Equine

quick question: lets say fetch_max_wait_ms time has elapsed and we still have not reached the min data byte size, what does the consumer do? Continue waiting or just accept the data that is already present? – Neighborhood

@Neighborhood Same question I have and second thing what is difference between fetch.wait.max.ms and the time we provide in consumer.poll(<what is this time>)? Is there anything related between them. – Kerry

The property is 'fetch.max.wait.ms' instead of 'fetch.wait.max.ms' atleast in Kafka version 2.3+ – Gemina