Docker Compose circular container linking

Question

I'm in the process of attempting to containerize our development environment with docker. This includes a mix of Google Appengine projects as well as services that are being eventually hosted in Google Compute engine within a containerized vm.

Our current development environment bootstrapping scripts bring up elasticsearch and nginx within boot2docker and the other applications run on localhost:{product port} within the dev_appserver appengine sandbox. This process is proving hard to manage and maintain as it takes a lot of insight into how our applications communicate.

I'm getting an error with docker-compose that is detecting a circular dependency between containers.

Circular import between cs and vbc and aa and sr.

As this configuration is only for development environments (mac osx), does anyone have an suggestions or ideas on a different approach to take when linking all of the product suites dependencies together.

A portion of docker-compose.yml:

elasticsearch:
  build: ./compute/containers/elasticsearch/elasticsearch
  ports:
    - "9200:9200"
  environment:
    - PROJECT_ID=localhost
nginx:
  build: ./compute/containers/elasticsearch/nginx
  links:
    - elasticsearch:localhost
  ports:
    - "9201:9201"
cs:
  build: ./CS
  command: dev_appserver.py /src/ --host=0.0.0.0 --admin_host=0.0.0.0 --port=8080 --admin_port=9080 --storage_path=/data/
  ports:
    - "8080:8080"
    - "9080:9080" 
  volumes:
   - /Users/source/CS/src:/src
   - /Users/source/CS/data:/data 
aa:
  build: ./AA
  command: dev_appserver.py /src/ --host=0.0.0.0 --admin_host=0.0.0.0 --port=8081 --admin_port=9081 --storage_path=/data/
  links:
    - vbc:vbc-local
    - st:st-local
    - elasticsearch:localhost    
  ports:
    - "8081:8081"
    - "9081:9081" 
  volumes:
   - /Users/source/AA/src:/src
   - /Users/source/AA/data:/data 
vbc:
  image: google/cloud-sdk
  command: dev_appserver.py /src/ --host=0.0.0.0 --admin_host=0.0.0.0 --port=8082 --admin_port=9082 --storage_path=/data/
  links:
    - cs:cs-local
    - sr:sr-local
    - sm:sm-local
    - ms:ms-local
    - st:st-local    
    - cis:cis-local
    - elasticsearch:localhost
  ports:
    - "8082:8082"
    - "9082:9082" 
  volumes:
   - /Users/source/VBC/src:/src
   - /Users/source/VBC/data:/data    
sr:
  build: ./SR
  command: dev_appserver.py /src/ --host=0.0.0.0 --admin_host=0.0.0.0 --port=8083 --admin_port=9083 --storage_path=/data/
  links:
    - cs:cs-local  
    - aa:aa-local      
  ports:
    - "8083:8083"
    - "9083:9083" 
  volumes:
   - /Users/source/SR/src:/src
   - /Users/source/SR/data:/data 

Ariminum

You might consider using a service registry like consul github.com/progrium/docker-consul and an automatic registrator for your containers like registry.hub.docker.com/u/progrium/registrator. With these components you could register your containers in consul and every container would have to read the needed information from consul, eliminating the circular dependencies. Adds more complexity though. – Genius

Answer 1

You should be able to use the following solution soon.

The circular linking is being fixed in PR # 1676

This is how they're addressing the issue. Simply put, they're going to make the containers able to speak to each other without linking. I've added the updates to the Docker Compose documentation below:

Networking in Compose

By default, Compose sets up a single default network for your app. Each container for a service joins the default network and is discoverable via DNS under the service's name.

Note: Your app's network is given the same name as the "project name", which is based on the name of the directory it lives in. See the CLI docs for how to override it.

For example, suppose your app is in a directory called myapp, and your docker-compose.yml looks like this:

web:
  build: .
  ports:
    - "8000:8000"
db:
  image: postgres

When you run docker-compose up, the following happens:

1. A network called myapp is created.
2. A container is created using web's configuration. It joins the network myapp under the name web.
3. A container is created using db's configuration. It joins the network myapp under the name db.

Each container can now look up the hostname web or db and get back the appropriate container's IP address. For example, web's application code could connect to the URL postgres://db:5432 and start using the Postgres database.

Because web explicitly maps a port, it's also accessible from the outside world via port 8000 on your Docker host's network interface.

Further reading on the experimental Docker networking API: https://github.com/docker/docker/blob/master/experimental/networking_api.md

Ine

Awesome, looking forward to trying this out! – Ariminum

Answer 2

Now with v2 docker-compose files definition, all the services are available between them without the need of a link section.

You can directly make a request to the service name from everyone to everyone (including a service to itself). So if you wanted to make a request from cs to vbc, you just curl vbc.

It's also possible to define a service with a custom domain name declaring a hostname key in the service section of the docker-compose file.

If you want to see more, the networking api is no longer experimental: https://github.com/docker/compose/blob/master/docs/networking.md

This is your docker-compose file in v2 without unnecessary links:

version: '2'

services:
  elasticsearch:
    build: ./compute/containers/elasticsearch/elasticsearch
    ports:
      - "9200:9200"
    environment:
      - PROJECT_ID=localhost
  nginx:
    build: ./compute/containers/elasticsearch/nginx
    ports:
      - "9201:9201"
  cs:
    build: ./CS
    command: dev_appserver.py /src/ --host=0.0.0.0 --admin_host=0.0.0.0 --port=8080 --admin_port=9080 --storage_path=/data/
    ports:
      - "8080:8080"
      - "9080:9080" 
    volumes:
     - /Users/source/CS/src:/src
     - /Users/source/CS/data:/data 
  aa:
    build: ./AA
    command: dev_appserver.py /src/ --host=0.0.0.0 --admin_host=0.0.0.0 --port=8081 --admin_port=9081 --storage_path=/data/
    ports:
      - "8081:8081"
      - "9081:9081" 
    volumes:
     - /Users/source/AA/src:/src
     - /Users/source/AA/data:/data 
  vbc:
    image: google/cloud-sdk
    command: dev_appserver.py /src/ --host=0.0.0.0 --admin_host=0.0.0.0 --port=8082 --admin_port=9082 --storage_path=/data/
    ports:
      - "8082:8082"
      - "9082:9082" 
    volumes:
     - /Users/source/VBC/src:/src
     - /Users/source/VBC/data:/data    
  sr:
    build: ./SR
    command: dev_appserver.py /src/ --host=0.0.0.0 --admin_host=0.0.0.0 --port=8083 --admin_port=9083 --storage_path=/data/
    ports:
      - "8083:8083"
      - "9083:9083" 
    volumes:
     - /Users/source/SR/src:/src
     - /Users/source/SR/data:/data 

Botfly

Answer 3

It is in your linking:

sr requires aa
aa requires vbc
vbc requires sr
sr requires aa

sr requires cs
sr requires vbc
vbc requires sr
vbc requires cs

you can see how this is circular.

Byars

Yes, I'm aware of this issue is because of the linking. However, in my single docker-compose file to "rule them all" for development environment bootstrapping this is a desired evil. The intent of the question was to figure out how to accomplish this. – Ariminum

Why do you want to link them together? Is it for communication? If so, then you have several options: 1. ambassador containers 2. proxy in between (I prefer this one personally) 3. several other projects that attempt to link remote containers. – Byars

Answer 4

What you want to do just don't links 2 containers to each other than cycle found error will come i have solved.

Example:

mongo: # container image name
 image: mongo:latest
 ... 
 ...
springboot-app:
 image: anythink
 ...
 ...
 links:
 - mongo #this image mongo container image name
angular-app:
 image: anythink
 ...
 ...
 links:
 - springboot-app

Pickup

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Nutpick