Zookeeper in AWS ECS

Question

I'm trying to launch a zookeeper cluster in AWS using Container Services. Using the base "zookeeper" docker image. There are two issues I'm currently having.

1) Zookeeper requires an "ZOO_MY_ID" variable to be set. When I run at home this variable is populated as numbers 1, 2, etc in the docker compose file. How can I set environment variables in ECS that would be incremented for every task started?

2) Zookeeper node needs to know its peers. It is set by "ZOO_SERVERS= server.1=10.0.0.1:2888:3888 server.2=10.0.0.2:2888:3888" variable. With the ip addresses assigned randomly how should I configure the zookeeper node so that it would discover its peers?

Thanks, Ruben

Foltz

Why do you plan to host Zookeeper in ECS? What is the usage scenario? – Sass

I plan to use the Zookeeper for cluster coordination and configuration for my microservices. Every service node registers itself in the zookeeper, discovers other service nodes, performs quorum and leadership election, etc. Once the zookeeper is running the rest of the pieces will get together. Now need to get the zookeeper cluster up and running, and also find a reliable way to discover endpoints of the zookeeper nodes. – Foltz

Are your Microservices also running in ECS? – Sass

Not yet, but yes, they are all going to be running in ECS. – Foltz

Here some functional overlaps are there with AWS ECS and Zookeeper. ECS itself can guarantee HA of containers with service abstraction and Load Balance using AWS Application Loadbalancer. Which is not coming by default is the the service discovery, which could be achived with AWS DNS service with custom triggers. Check the following reference architecture aws.amazon.com/blogs/compute/… if you have provisions to solve the problems without using Zookeeper. – Sass

The problem I'm solving requires knowledge about individual nodes. A simple Load Balancing wouldn't work. I've seen that article. It doesn't provide complete solution for what I'm trying to do. But the direction seems reasonable. I was just trying to find an out of the box solution to run Zookeeper in ECS. – Foltz

Did you somehow resolve this? I am about to hit the same issue you've described. – Hobard