What is Consumer Group in azure Event Hub?

Question

Can anyone please figure out what is Consumer Group in Azure Event Hubs. And What is the use of it? I have surfed a lot of sites but I can't get a clear answer.

Aubervilliers

Answer 1

From the docs:

Consumer groups: A view (state, position, or offset) of an entire event hub. Consumer groups enable consuming applications to each have a separate view of the event stream. They read the stream independently at their own pace and with their own offsets.

Diagram:

According to this consumer groups are the way to logically separate your consumers, so they only see the events they are interested in.

Squamulose

Thank you @4c74356b41. Now I could understand the consumer groups – Aubervilliers

how can we group several subscriptions or event listeners under a consumer group in Azure? – Aubervilliers

You can create up to 20 consumer groups for a Standard tier event hub. There can be at most 5 concurrent readers on a partition per consumer group; however, it is recommended that there is only one active receiver on a partition per consumer group. Within a single partition, each reader receives all of the messages. If you have multiple readers on the same partition, then you process duplicate messages. You need to handle this in your code, – Accordingly

does it allows to read messages existed before creating new consumer group? – Wristwatch

Consumer groups get ALL events, not just the ones they're interested in: #49657456 – Youth

Found this thread while looking for information on Eventhub consumer-group. One follow-up question - Where is the offset stored ? at the client side (Event receiver side) or with the Eventhub server side. – Hero

Answer 2

Think of a consumer group as a label that groups one or more event consumers together as a set. It's often named after what responsibility the consumer has in an application (ex: "Telemetry", "OrderProcessing"). A default consumer group named "$Default" is created when an Event Hub is created.

Like mentioned in the previous answer, consumer groups allow multiple applications to have their own view of an event stream, thus allowing each to read the stream independently and at their own speed. For example, you might have a downstream storage writer application that writes event data to long-term storage, and then another application that performs complex event processing--the two applications would belong to two different consumer groups.

Since an Event Hubs consumer is associated with a specific Event Hubs and consumer group, if you pass in the same consumer group as a parameter when constructing EventHubConsumerClients, then those clients will be associated with the same consumer group (thus grouping the event consumers):

var consumer = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);

Note that you can only have multiple consumer groups if you're using the Standard tier service.

Aston

Answer 3

In Azure Event Hub, consumer groups are a means of limiting the number of concurrent readers to the same partitions.

They serve no real purpose except perhaps a practical one which is to guide you away from reading the same events multiple times from the same system or application.

There's also a pricing aspect which is that to enable multiple applications subscribing to the same event stream, you may need to upgrade to a better plan (for example, the basic plan allows just a single consumer group).

As others have pointed out, the documentation explains consumer groups as a "view of an entire event hub". This view, however, has no practical bearing.

You could run 5 different applications on a single consumer group, but it is impractical perhaps, because this won't scale to the 6th application should you ever need it since there is a limit of 5 concurrent readers to the same partition per consumer group.

Comparison to Kafka consumer groups

Event Hub consumer groups are entirely different than Kafka consumer groups (whereby offsets are stored automatically as you progress on reading from the topic).

Clarification here: https://github.com/Azure/azure-event-hubs-for-kafka#more-faq.

When using the Kafka API, Event Hubs do support Kafka consumer groups.

Tanjatanjore