Can single consumer read from multiple partitions of a kafka topic?

Question

There is a kafka topic with 16 partitions

With a given consumer group name, we are currently launching single consumer to read from the topic.

---

1. Does single consumer read from partition 0(only) of that topic? If partition 0 has gone empty with messages, does consumer start reading from next partition(partiton 1... and so on)?

2. we have option to launch more than consumer(with same consumer group name) to read from same topic(having 16 partitions). How many consumers can be maintained, to read from multiple partitions parallely?

Simferopol

Answer 1

The consumers in a group divide the topic partitions as fairly amongst themselves as possible by establishing that each partition is only consumed by a single consumer from the group. When the number of consumers is lower than partitions, same consumers are going to read messages from more than one partition.

In your scenario, a single consumer is going to read from all your partitions. This type of consumer is known as exclusive consumer. This happens when consumer groups have only one consumer. Such a consumer must be connected to all partitions it requires.

Ideally, the number of partitions should be equal to the number of consumers. Should the number of consumers be greater, the excess consumers were to be idle, wasting client resources. If the number of partitions is greater, some consumers will read from multiple partitions, which should not be an issue unless the ordering of messages is important.

Ordering

Kafka does not guarantee ordering of messages between partitions. It does provide ordering within a partition. Therefore, Kafka can maintain message ordering for a consumer if it is subscribed to only a single partition.

If message ordering is required in your use case, the messages send from producer should be using a same partition key to be grouped into same partition in kafka broker.

Mohandas

Answer 2

you have a topic with 16 partitions

With a given consumer group name, you are currently launching single consumer to read from the topic This single consumer will read from all partitions.

Since you can launch multiple consumer (within same consumer group).

For 16 partitions you should have max 16 consumer listening to each partitions. In Kafka within a consumer group you can have max 1 consumer per partition.

Measurement