How to use Android App as a client for Kafka?

Question

Is it possible / does it make sense to use an Android app as a "Producing client" for Apache Kafka?

Let's say my Android App need to capture and analyse reaction time data. Goal is to collect all data and show the average reaction time in real-time in the App.

The alternative is having an app server of some kind as an intermediary that accepts messages from the android app and posts them to Kafka, rather than having the app be a Kafka Producer on its own.

Heteronomous

Answer 1

Even if it's possible, in my opinion it has some disadvantages.

In general I like clients to be as simple as possible to avoid maintenance issues. Instead I'd route all client requests through a REST API on my app server. The disadvantages are not related to Kafka, but are common problems of native clients.

Coupling

You're coupling the Android app closely to your messaging infrastructure. If you later decide that a Kafka solution is too much and Plain Old Java would be good enough, you'll first have to update the Android app and wait until enough users do an update.

Network issues + delivery guarantees

Kafka clients also require a direct connection to each of the brokers. Mobile clients can have very inconsistent/spotty network connectivity, making direct client access susceptible to dropped events and overall network connectivity issues.

Authentication

Probably you already have some kind of authentication in your app. You can also create authenticated connections to Kafka. So you'll have two authentication paths, whereas with an app server Kafka only needs to check if the requests are coming from the trusted app server, which means less implementation effort.

...

Damper

Of coure there might be use cases where connection Kafka directly is the better option. But you didn't explain for example why you need to send the data at all to the server, so I can only give a general answer. – Damper

thanks so much for your opinion! so you say it's possible. do you have any code snippets or useful links for this approach? – Heteronomous

@lidox: Using Kafka in Android isn't different than in any other Java application. I did not try it but there shouldn't be any compatibility problems. Of course you have other challenges. For example there isn't a permanent internet connection so that you must store the messages temporarily on the device and retry sending them later on. If you have any specific problems please create another question and post me a link here. – Damper

Ok here we go: #40448689 – Heteronomous

@ChristianStrempfer You're saying having the web server act like consumer of kafka? And When the consumer receives a message from kafka, have that message send to the app? – Leasehold

@SKSV This question is about producing messages not receiving, but the same arguments apply for consuming. Let the web server consume the messages and use the standard way of the mobile ecosystem to sent notifications. – Damper

"Using Kafka in Android isn't different than in any other Java application"... This is false; please try before making such claims. Android SDK does not run the same Java bytecode as the JRE necessary to run Kafka clients – Hiddenite

This question deserves many points as it gives an insight to architecture designer – Cowell

All kafka clients use the librdkafka C Library under the hood which implements the apache kafka protocol github.com/confluentinc/librdkafka Check the installation section in the link, it doesn't mention android! It mentions Mac OSX, Debian, Ubuntu, RedHat, CentOS, Fedora and windows – Interest

Even if kafka client could run on android I still think this is not a good mobile solution! Other frameworks and products exist that provide you with WebSocket based real time bidirectional data synchronization between mobile clients and the backend e.g. couchbase mobile, mongodb + realm and microsoft sync framework – Interest

@Interest Fully agree. I hope I made that clear in the first paragraph. – Damper

Answer 2

I think it would make lots of sense:

Kafka-clients.jar provides auto-reconnecting capability which is very useful when the phone is on a flaky internet connection. It is also quite thin and does not include any of Kafka Server code (it doesn't even depend on Scala).

Unfortunately, it isn't compatible with Android just now: KAFKA-7025 . If you'd like to see this happen, please upvote the JIRA issue.

Alialia

Answer 3

I believe that an Android Application can use a secured connection to a Kafka Broker cluster using SASL, for example. However, it must be done in coupling with any other communication, which can support synchronized keys rotation, which is initialized by any remote server with synchronization with the broker's cluster. Any concrete instance of mobile application can listen to a concrete topic, and produce messages to a related topic, which is created when registering the instance using a REST server. Any deserializer verifies headers or Keys for a token, which are appointed using REST while registering on the same service. Custom encryption can be provided similarly. Technically it is solvable. But what are the benefits, in front of using Firebase, for instance? Expenses I see from the start. Benefits???

Bramble

No Android app can use Kafka client since core JVM libraries such as java.management don't exist in Android. I'm not sure I understand your point about topics and REST. Plus, Firebase is a SaaS tool and not everyone uses/needs it; it's not a replacement for pubsub message queue – Hiddenite