Connect two instances of docker-compose [duplicate]

Question

I have a dockerized application with a few services running using docker-compose. I'd like to connect this application with ElasticSearch/Logstash/Kibana (ELK) using another docker-compose application, docker-elk. Both of them are running in the same docker machine in development. In production, that will probably not be the case.

How can I configure my application's docker-compose.yml to link to the ELK stack?

Cetus

You can't. you either need to use your host machine network to connect to apps or use overlay networking with you docker compose which I believe still not mature enough for production use cases. – Apivorous

Answer 1

Update Jun 2016

The answer below is outdated starting with docker 1.10. See this other similar answer for the new solution. https://mcmap.net/q/442470/-how-to-link-docker-container-to-each-other-with-docker-compose

Old answer

1. Create a network:

    $ docker network create --driver bridge my-net
   

2. Reference that network as an environment variable (${NETWORK})in the docker-compose.yml files. Eg:

   pg:
     image: postgres:9.4.4
     container_name: pg
     net: ${NETWORK}
     ports:
       - "5432"
   myapp:
     image: quay.io/myco/myapp
     container_name: myapp
     environment:
       DATABASE_URL: "http://pg:5432"
     net: ${NETWORK}
     ports:
       - "3000:3000"
   

Note that pg in http://pg:5432 will resolve to the ip address of the pg service (container). No need to hardcode ip addresses; An entry for pg is automatically added to the /etc/host of the myapp container.

3. Call docker-compose, passing it the network you created:

   $ NETWORK=my-net docker-compose up -d -f docker-compose.yml -f other-compose.yml
   

I've created a bridge network above which only works within one node (host). Good for dev. If you need to get two nodes to talk to each other, you need to create an overlay network. Same principle though. You pass the network name to the docker-compose up command.

Bandog

Answer 2

You could also create a network with docker outside your docker-compose :

docker network create my-shared-network 

And in your docker-compose.yml :

version: '2'
services:
  pg:
    image: postgres:9.4.4
    container_name: pg
    expose:
      - "5432"
networks:
  default:
    external:
      name: my-shared-network

And in your second docker-compose.yml :

version: '2'
  myapp:
    image: quay.io/myco/myapp
    container_name: myapp
    environment:
      DATABASE_URL: "http://pg:5432"
    net: ${NETWORK}
    expose:
      - "3000"
networks:
  default:
    external:
      name: my-shared-network

And both instances will be able to see each other, without open ports on host, you just need to expose ports, and there will see each other through the network : "my-shared-network".

Lamere

Is net: ${NETWORK} required in the second docker-compose.yml? – Astronautics

If one of the service has to start before another, how can we implement link or depends_on. – Fritzfritze

Answer 3

I didn't find any complete answer, so decided to explain it in a complete and simple way.

To connect two docker-compose you need a network and putting both docker-composes in that network, you could create netwrok with docker network create name-of-network,

or you could simply put network declaration in networks option of docker-compose file and when you run docker-compose (docker-compose up) the network would be created automatically.

put the below lines in both docker-compose files

networks:
   net-for-alpine:
     name: test-db-net

Note: net-for-alpine is internal name of the network and it will be used inside of the docker-compose files and could be different, test-db-net is external name of the network and must be same in two docker-compose files.

Assume we have docker-compose.db.yml and docker-compose.alpine.yml

docker-compose.apline.yml would be:

version: '3.8'

services:

  alpine:
    image: alpine:3.14
    container_name: alpine
    networks:
      - net-for-alpine
  
    # these two command keeps apline container running
    stdin_open: true # docker run -i
    tty: true # docker run -t



networks:
  net-for-alpine:
    name: test-db-net

docker-compose.db.yml would be:

version: '3.8'

services:

  db:
    image: postgres:13.4-alpine
    container_name: psql
    networks:
      - net-for-db
  
networks:
  net-for-db:
    name: test-db-net

To test the network, go inside alpine container

docker exec -it alpine sh 
      

then with following commands you can check the network

# if it returns 0 or see nothing as a result, network is established
nc -z psql (container name)  

or

ping pgsql

Battle

very useful, but you have to fix the ping pgsql into ping psql and moreover the second docker compose needs the environment variablePOSTGRES_PASSWORD: "password". I can't edit your answer because of StackOverflow limits unfortunately – Thomey

Answer 4

If you set a predictable project name for the first composition you can use external_links to reference external containers by name from a different compose file.

In the next docker-compose release (1.6) you will be able to use user defined networks, and have both compositions join the same network.

Padron

Answer 5

Take a look at multi-host docker networking

Networking is a feature of Docker Engine that allows you to create virtual networks and attach containers to them so you can create the network topology that is right for your application. The networked containers can even span multiple hosts, so you don’t have to worry about what host your container lands on. They seamlessly communicate with each other wherever they are – thus enabling true distributed applications.

Leung

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review – Springhalt

@tmoreau I agree. But this question is too broad, the topic too complex, to give a short answer. I removed the link so OP can google. – Leung

If the question is too broad, flag as such. Your answer could be a comment. – Springhalt

That's a matter of opinion. The answer is too broad, not the question. Answers are for adding solutions to the question. Comments to add essential information, ask for clarification, ask for extra information, etc. Although short the answer may serve as a future reference. – Leung