Messaging vs RPC in a distributed system (Openstack vs K8s/Swarm)

Question

OpenStack uses messaging (RabbitMQ by default I think ?) for the communication between the nodes. On the other hand Kubernetes (lineage of Google's internal Borg) uses RPC. Docker's swarm uses RPC as well. Both are gRPC/protofbuf based which seems to be used heavily inside Google as well.

I understand that messaging platforms like Kafka are widely used for streaming data and log aggregation. But systems like OpenStack, Kubernetes, Docker Swarm etc. need specific interactions between the nodes and RPC seems like a natural choice because it allows APIs to be defined for specific operations.

Did OpenStack chose messaging after evaluating the pros and cons of messaging vs RPC? Are there any good blogs/system reviews comparing the success of large scale systems using messaging vs RPC? Does messaging offer any advantage over RPC in scaled distributed systems?

Selfridge

I'll note there's a small hint of a false dichotomy. In an RPC world there's still use for message queues, like pubsub for gRPC. But they aren't the only option. – Disobedience

Answer 1

Does messaging offer any advantage over RPC in scaled distributed systems ?

Mostly persistence is a big advantage for messaging system. Another point is broadcasting. You need to implement this into gRPC by yourself. Service Discovery and Security can be another reason. In Messaging System you just need to keep one system highly secure, while with gRPC you might have many points where somebody could break into the system. Message queue systems usually already have some kind of service discovery implemented. With gRPC you have to use at least another library for this.

Are there any good literate comparing the success of large scale systems using messaging vs RPC ?

It's not a vs. There are different use cases. Messaging Systems are generally slower than RPC protocols. Not only slower than gRPC. The reason for this is also simple. You just introduce a middleware between two or more nodes. But they provide persistence, broadcasting, Pub/Sub etc.

Did Openstack chose messaging after evaluating the pros cons of messaging vs RPC ? Probably

Does messaging offer any advantage over RPC in scaled distributed systems ?

1. Ready to use solution, just use a client
2. Persistence
3. Ready to use Service Discovery
4. Pub/Sub Pattern
5. Failure tolerance

Most of the points needs to be implemented with gRPC by yourself.

Rushing