Kafka Consumer default Group Id

Question

I'm working with Apache Kafka and its Java client and I see that messages are load balanced across different Kafka Consumers belonging to the same group (i.e. sharing the same group id).

In my application I need all consumers to read all messages.

So I have several questions:

• if I don't set any group id in the Consumer Properties, what group id will the Kafka Consumer be given?

• Is there a single default value?

• Does the client create a random value each time?

• Do I need to create a different id for each consumer to be sure that each one receives all messages?

EDIT: Thank you for your answers.

You are correct: if one doesn't set the consumer group id, Kafka should complain.

However, I have found out that if the group id is null, the Java client sets it to the empty string "" to avoid problems. So apparently that is the default value I was looking for.

Surprising all my consumers, even if I don't set their groupIds (and so they are all with groupId == "") seem to receive all the messages the producer writes.

I still can't explain this: any suggestions?

Pentagram

It's probably linked to your number of partitions. How many partitions do you have and how many consumers do you use? In fact, when multiple consumers are subscribed to a topic and belong to the same consumer group, then each consumer in the group will receive messages from a different subset of the partitions in the topic. – Widower

I use a single partition for each topic, so all consumers receive messages from the same partition :/ – Pentagram

Answer 1

if I don't set any group id in the Consumer Properties, what group id will the Kafka Consumer be given?

The kafka consumer will not have any consumer group. Instead you will get this error : The configured groupId is invalid

Is there a single default value?

Yes, you can see the consumer.properties file of kafka for reference. The default consumer group id is: group.id=test-consumer-group

Does the client create a random value each time?

No, groupId seems to be mandatory for Java client starting Kafka 0.9.0.x consumers. You can refer to this JIRA: https://issues.apache.org/jira/browse/KAFKA-2648

Do I need to create a different id for each consumer to be sure that each one receives all messages?

Yes, if all consumers use the same group id, messages in a topic are distributed among those consumers. In other words, each consumer will get a non-overlapping subset of the messages. Having more consumers in the same group increases the degree of parallelism and the overall throughput of consumption. On the other hand, if each consumer is in its own group, each consumer will get a full copy of all messages.

Widower

What is the difference between the default console-consumer-* that is created when we run the kafka-console-consumer without any consumer Group and the one mentioned in the consumer-properties ``` # consumer group id group.id=test-consumer-group ``` – Sunbeam

When using kafka-console-consumer without specifying a consumer group, it operates as a standalone consumer. It reads messages from the topic, starting from the earliest or latest offset. This consumer doesn't belong to any specific consumer group and doesn't have group coordination features like load balancing or partition reassignment. On the other hand, when a consumer specifies a group.id in the consumer properties (like in the example you provided group.id=test-consumer-group), it becomes part of a consumer group. – Widower

Answer 2

Don't want to repeat other answers, but just to point out something: You don't actually need a consumer group to consume all messages. The Kafka Consumer API (assuming we're dealing with the Java one) has both a subscribe() and an assign() method. If you want all consumers to receive all messages without load balancing (which is what essentially consumer groups are for), you can just invoke assign() on all consumers, passing it all the partitions for the topic, optionally followed by seek() to set the offsets; that way your consumers will get all messages.

This way Kafka will not manage partition assignment and will not persist offsets — the consumer is responsible for all that. Depending on your use case, it may be a better approach comparing to having a consumer group per consumer.

Cogan

I had to implement this and was looking for examples, found one to understand better mbukowicz.github.io/kafka/2020/09/12/….. – Depredate

Answer 3

I have same problem. And took some time to research this issue. The project spring-cloud-stream will check whether you have set the group id for consumer. If not, spring-cloud-stream will create a random value as group id. Please refer the method createConsumerEndpoint in class KafkaMessageChannelBinder.

Ebarta

Answer 4

According to KIP-289 the default group.id has been "improved" and the default group.id, since kafka clients version 2.2.0, is null.

KIP-289: Improve the default group id behavior in KafkaConsumer.

It seems to me that when using assign you can forgo the group.id, leaving it to null, and there will be no offsets available.

Fanni

Answer 5

Check the groupId from- @KafkaListener(topics = "${kafka.topic}", groupId = "groupIdName")

Steps-> Go to Kafka folder

Open config folder

Open consumer.properties

Change the group id

group.id=groupIdName

Partake

Answer 6

If don't set group.id,you will get error when consume topic data.

org.apache.kafka.common.errors.InvalidGroupIdException: The configured groupId is invalid
22:08:14.132 [testAuto-kafka-consumer-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining group 
22:08:14.132 [testAuto-kafka-consumer-1] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending JoinGroup ({group_id=,session_timeout=15000,rebalance_timeout=300000,member_id=,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0 lim=18 cap=18]}]}) to coordinator bogon:9092 (id: 2147483647 rack: null)
22:08:14.132 [testAuto-kafka-consumer-1] ERROR org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Attempt to join group  failed due to fatal error: The configured groupId is invalid
22:08:14.132 [testAuto-kafka-consumer-1] ERROR org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer - Container exception

Vizcacha

Answer 7

With this exception missing groupId,

The library says that the default is an empty string but this did not work for me. What worked for me was rather a single space between quotes i.e groupId =" " and not groupId=""

Toolmaker

Answer 8

if you are using nodejs like me. You can check if clientId is empty.

const kafka = new Kafka({ clientId, brokers })
const consumer = kafka.consumer({ groupId: clientId })

Volga