Dynamic Scalable and adaptive architecture

Question

I am a PhD student in Cloud Computing, I plan to use the microservices based architecture with consul and zeromq for my research project. I had few questions that I am finding hard to understand. Can someone help me out in sharing their experience.

1. We have microservices based on dockers, We have zeromq and we have consul. Can you mention how we could combine all the three together to have a dynamic adaptive environment?

Though I understand as to what zeromq, docker and consul is individually, I am still unable to get a clear picture of how all of them function as a whole.We have docker containers having microservices running inside them on a host. We use zeromq for transport (Pub-sub/pipeline) of messages between docker containers. The containers may be running on the same host/datacenter or on different hosts/datacenters. We then use consul for service discovery.Is my understanding correct here?

2. How does the architecture dynamically scale up/down according to workload?

Say, I have a situation where I need more worker nodes for a particular computation for sometime. Who spins up more number of worker nodes. Which component determines/takes this decision?

Is there a scheduling component? If so, can someone briefly explain how it happens or which component performs that function?

3. So, what is the major role of consul? Is it used just for service discovery?Can it be used for configurations as well. If so, whats its limitation?

I see that even zeromq has service discovery mechanisms, so why do we require consul?

4. How does a failure of a node information gets propagated in the architecture? Which component is responsible? Is it just consul ? Or zeroMq also?

Please advice.

Mcewen

Answer 1

I am involved in a large project using Docker-based microservices and Consul. (We are using a different queueing service-- RabbitMQ, so I can't speak in details to that aspect, but in general, a queue is a queue.)

Neither Docker, Consul, nor any queue technology that I am aware of provide autoscaling functionality. Docker provides an easy way to spin up multiple instances of the service, and Consul provides service discovery (as you said) and a key/value persistence store. The queue is just a way of passing messages between the service instances. There's nothing you've mentioned that handles autoscaling.

To add autoscaling functionality, you'll need to look at something like Kubernetes.

You could look at something like CloudFoundry or Mesos. However, both of those require a virtualization layer, such as OpenStack or VMWare vSphere. With those products, comes significant value, but also price and complexity.

Instead of going that route, I recommend you look into Amazon Web Services. Using AWS, you can easily run docker containers and set up autoscaling based on CPU load, messages in the queue, time of day (or day of week), etc. I know using AWS carries a price tag, but when well designed and managed, it'll cost you far less than attempting to design, implement and maintain yourself. You can also use the smallest (ie, free) machines and or spot instances to minimize costs.

Vasodilator

I'm pretty sure ASGs do not support autoscaling ECS or vanilla Docker containers on EC2 directly. Kubernetes and Mesosphere do. CSP-provided Container support is fairly limited outside of Joyent's Triton right now. #29737534 – Acanthopterygian

Answer 2

This is an interesting question. All of the components you mention are independent with clearly separate strengths and roles in a microservice architecture. The unusual part is in using messaging rather than HTTP. I think it's a valuable departure since it enables more flexible computing patterns (work producer doesn't need to be product consumer or even be notified). A particular beauty of skipping HTTP is avoiding the cost (both OPEX and service latency), complexity, and added fault modes of load balancers.

1. Docker: to manage the packaging of individual services and delivery onto infrastructure ZeroMQ: to manage efficient peer-to-peer or brokered communications between services Consul: service discovery (ex. find out where the user service is)

2. Autoscaling isn't performed by Docker. You could do this with your own microservice that inspects the load metrics (ex. load average, phys/swap memory usage, etc.) and spins up replicas and updates Consul.

   Alternatively, you could lean on autoscaling solutions detailed by @drhender: Kubernetes, Mesosphere DCOS, or AWS Autoscaling Groups. Note, however, that using AWS Autoscaling Groups signicantly limits the portability of your solution.

   Whatever autoscaling mechanism you choose, make sure your ZeroMQ message patterns support services being added or removed. The ZeroMQ Guide has good advice on this topic.

3. Consul can provide both service discovery and service configuration needs. Bear in mind storage security concerns if you are storing sensitive data such as PHI or PII. Those are better stored with at-rest protections such as Vault supplies.

   The Consul agent collects telemetry that you can monitor for issues. There is also a fairly simple Consul KV benchmarking suite that you can use to test the configuration storage.

   ZeroMQ isn't designed for service discovery directly. You can choose a centrally brokered architecture that makes separate service discovery unnecessary, but that has different scalability and fault tolerance implications. It has a per-message payload overhead assuming you use multi-part messages to and topic subscriptions when binding SUB sockets. There are many ways you could do service discovery with ZeroMQ alone, but it would be non-trivial particularly when factoring in faukt-tolerance and consensus.

4. Node failure is an interesting challenge. Consul could know via health checks, but ZeroMQ creates some challenges depending on messaging patterns. For example, using a REQ-REP pair if a request is sent and the responder dies after delivery the REQ socket will block forever in my experience. There are ways around this with timeouts.

   I would lean on Consul for this and be prepared to interrupt or respawn REQ sockets on failures. Avoiding RPC style interactions entirely by using a Staged Event Driven Architecture (SEDA) where producers of inputd are not the consumers of outputs side-steps this almost entirely. You always have the challenge of losing the queued inputs or outputs on failure so you need system level monitoring and retry mechanisms if losing work is fatal.

ContainerBuddy allows you to put any launchable application in a Docker container and have it register with Consul. It could simplify things for you.

Credulity