One to One and Group Messaging using Kafka

Question

As Kafka has a topic based pub-sub architecture how can I handle One-to-One and Group Messaging part of web application using Kafka? I am using SpringBoot+Angular stack and Docker Kafka server.

Seminar

Kafka supports one-to-one delivery, you need to provide more details – Cavy

@Cavy I have a web app of SpringBoot+Angular stack and there are user registration and others. I want to add messaging part among the users using kafka but I am confused how to do so using kafka topic and partitions – Seminar

Answer 1

I'll write another answer here. Based on my experience with the chatting service. You only need one topic for all the messages. Using a well designed Message body.

public class Message {
 private String from; // user id
 private String to;  // user id or group id
}

Then you can create like 100 partitions for this topic and create two consumers to consume them (50 partitions for one consumer in the beginning). Then if your system reaches the bottleneck, you can easier scale X more consumers to handle the load.

How to do distribute the messages in the consumer. I used to send the messages to the Mobile app, so all the app has a long-existing connection to the server, and the server sends the messages to the app by that channel. For group chat, I create a Redis cache to store all the active users in the group, so I can easier get the users who belong to this group, send them the messages.

And another thing, Kafka stateless, means Kafka doesn't de-coupled from the business logic, only acts as a message system, transfers the messages. If you connect your business logic to Kafka, like create a topic "One-to-One" and delete some after they finished, Kafka will be very messy.

Cavy

yeah.. Thanx. I understand implementing business logic like delete/edit messages is messy here.. but can I do this actually? so you are suggesting to create a consumer just to consume all messages and then route them in backend according to message addresses? can u explain a little bit about using redis cache to manage groups? another thing is should I use separate database for some logic implementation purposes that with whom and which groups he has been active so I have to render those messages with group/sender names? – Seminar

What would be your partition key in this example? – Particulate

Answer 2

By One-to-One, I suppose you mean one producer and one consumer i.e. using at as a queue.

This is certainly possible with Kafka. You can have one consumer subscribe to a topic and and restrict others by not giving them authorization . See Authorization in Kafka

Note that once a message is consumed, it is not deleted, rather it is committed so that the same consumer will not consume it again.

By Group Messaging, I suppose you mean one producer > multiple consumers or multiple-producer > multiple-consumers

This is also possible, a producer can produce messages to a topic and multiple consumers can consume them.

If all the consumers have the same group id, then each consumer in the group gets only a subset of messages.

If they have different group ids then each consumer will get all messages.

Multiple producers also can produce to the same topic.

A consumer can also subscribe to multiple topics.

Marginal

yeah, that's mean I have to create topic for each user and delete topic when user gets deleted? same to group messaging? – Seminar

@Naiul, You don't delete messages in Kafka. Kafka will delete it follow it's configuration. Consumers only commit the offset, which means they consumed the message, it's time to move to the next one. – Cavy

@NafiulAlamFuji You don't need to create a topic for each user, rather store all the users in one topic where user_id could be the key of your message. Once the user gets deleted, simply pass null value for that user_id as the next message to that topic. – Marginal

@Marginal as far I understand, I have to create one topic for "One-to-One" and another for "Group" and then in "One-to-One" topic each partition will be belong to one user(subscribe) for incoming messages to him. And in topic "Group" each partition will be belong to group of people(subscribe) who want to chat together but not kafka user group as it allows no two process in same user group to consume messages at the same time. Is this OK? – Seminar

@NafiulAlamFuji I don't understand what you are trying to achieve i.e. for what purpose you are going to use Kafka.. Could you elaborate your use-case? – Marginal

my use case is "One-to-One" and "Group" messaging using kafka.. I meant chatting.. like messenger or whatsapp... that's why I am wanted to know how this use case can be achieved with integrating kafka features and springboot backend. – Seminar

Answer 3

Ok, It's a very complicated question, I try to type some simple basic information.

Kafka topics are divided into a number of partitions. Partitions allow you to parallelize a topic by splitting the data in a particular topic across multiple brokers — each partition can be placed on a separate machine to allow for multiple consumers to read from a topic in parallel.

So if you are using partitions, means you have multiple consumers to consume some in parallel.

consumer groups for a given topic — each consumer within the group reads from a unique partition and the group as a whole consumes all messages from the entire topic.

Basically, you can have only one group, then the message will not be processed twice in the same consumer group, and this is how Kafka delivers exactly once.

If you need two consumer groups, you need to think about why you need two? Are the consumers in two groups handling the different logic?

There is more, please check the official document, or you can answer a smaller question.

Cavy

as far I understand, I have to create one topic for "One-to-One" and another for "Group" and then in "One-to-One" topic each partition will be belong to one user(subscribe) for incoming messages to him. And in topic "Group" each partition will be belong to group of people(subscribe) who want to chat together but not kafka user group as it allows no two process in same user group to consume messages at the same time. Is this OK? – Seminar

Ok, a little confused when you choose "Group" as the topic name. But my description about Kafka still counts. For "One-to-One", you mentioned about one partition one consumer, that means you have consumers in the same group(the size should be equals to the partition size, so one consume will consume one partition) – Cavy

But how about topic "Group", it's very hard to understand about "chat together?", I mean I understand you want to create something like a chatting group, but maybe Kafka is not the best choice for this, because in this case your producer also have to be a consumer. People are sending messages while receiving messages. – Cavy

yeah, I am still confused about this use case. I think the One-to-One implementation with partition-To-user is ok. But will there be any problem if producer also becomes a consumer? because even while I am implementing One-to-One chat, if producer A wanna send message to B, Then A have to subscribe A partition, B have to subscribe B partition and A also should send messages to both B+A & B also should send messages A+B so that any chat (send+receive) between them will pop up in both A and B clients from kafka so sending messaging also stays? can kafka consumer filter/sort incoming messages ? – Seminar

@NafiulAlamFuji, I worked on IM app before. The solution I used is, build a message body with form(who sent it) and to(who will receive it), then I only need one topic to handle all the messages. It's consumer's job to dispatch all the messages to the right person or group. – Cavy

yeah.. I asked about this message routing to consumers with same architecture that where should I route this messages using kafka?(but that post was closed for being lengthy) because think about this,thousands of messages come to same topic and if I have to implement routing in spring boot according to "receiver" field in payload then I am not using kafka high throughput feature.. there should be a way where I can use kafka power in routing (like using each partition for each user and kafka will do the rest-jay kreps in quora) so,I am just seeking better ideas like this for planning – Seminar

@NafiulAlamFuji, better design your message and using Kafka topic in a stateless way, it'll be much easier when you need to scale. – Cavy

can u explain a little bit? how should I use kafka in stateless way and route messages in a kafka topic using spring backend? – Seminar

I try to explain this in another answer. You need more information, check GitHub more some use cases. – Cavy