Is it a good way to run Kafka on Kubernetes?

Question

For a large online application, use k8s to run it. The scale maybe daily activity user 500,000.

The application inside k8s need messaging feature - Pub/Sub, there are these options:

• Kafka
• RabbitMQ
• Redis

Kafka

It needs zookeeper and good to run on os depends on disk I/O. So if install it into k8s cluster, how? The performance will be worse?

And, if keep Kafka outside of the k8s cluster, connect Kafka from application inside the k8s cluster, how about that performance? They are in the different layer, won't be slow?

RabbitMQ

It's slow than Kafka, but for a daily activity user 500,000 application, is it good enough? If so, maybe it's a good choice.

Redis

It's another option. Maybe the most simple one. But from the internet I got that it will lose message sometimes. If true, that's terrible.

---

So, the most important thing is, use Kafka(also with zookeeper) on k8s, good or not in this use case?

Chandachandal

Answer 1

Yes, running Kafka on Kubernetes is great. Check out this example: https://github.com/Yolean/kubernetes-kafka. It includes ZooKeeper and Kafka as StatefulSets.

---

PS. Running any of the services in your question on Kubernetes will be pleasant. You can Google the name of the service and "kubernetes" and find example manifests. Many examples here: https://github.com/kubernetes/charts.

Cottager

Thank you very much for your answer. As I asked and worried, it the performance the same as use pure OS and on k8s with docker? – Chandachandal

I think there’s always overhead with kubernetes. I’ve not benchmarked it but I read kubernetes does come with overhead. I would google it. Personally, though, I find it the overhead can’t detract given the convenience. – Cottager

Answer 2

For Kafka, you can find some suggestion here. Kubernetes 1.7+ supports local persistent volume, which may be good for Kafka deployment.

Assailant

Answer 3

You can also take a look to the following project :

https://github.com/EnMasseProject/barnabas

It's about running Kafka on Kubernetes and OpenShift as well. It provides deploying with StatefulSets with persistent volumes or just in memory (for developing or just testing purpose). It provides deploying for Kafka Connect and Prometheus metrics as well.

Trowbridge

Answer 4

Another simple configuration of Kafka/Zookeeper on Kubernetes in DigitalOcean with external access:

https://github.com/StanislavKo/k8s_digitalocean_kafka

You can connect to Kafka from outside of AWS/DO/GCE by regular binary protocol. Connection is PLAINTEXT or SASL_PLAINTEXT (user/password).

Kafka cluster is StatefulSet, so you can scale cluster easily.

Hierarchy

Please don't add the same answer - 1, 2, 3, 4, - to multiple questions. Answer the best one and flag the rest as duplicates. See Is it acceptable to add a duplicate answer to several questions? – Lepanto