docker-compose: specify which interface will connect to each network (when having multiple networks)

Question

Inside the docker-compose.yaml I define 4 different networks:

networks:
  vpp-nw:
  vpp-client:
  vpp-server:
  vpp-db:

which use the following network addresses respectively:

172.20.0.x
172.21.0.x
172.22.0.x
172.23.0.x

One of the containers I use, connects to all 4 networks (with the same order):

# part of docker-compose.yaml

services:
  my_tool:
    build: ./my_tool
    image: tgogos/my_tool
    container_name: my_tool_docker_comp
    hostname: my_tool
    privileged: true
    links:
      - my_db
    networks:
      - vpp-nw
      - vpp-client
      - vpp-server
      - vpp-db
    ports:
      - "8888:8888"

Is there a way to define which interface will connect to each network? For example, I would like:

• eth0 to connect to the first (vpp-nw)
• eth1 to connect to the second (vpp-client)
• eth2 to connect to the third (vpp-server)
• eth3 to connect to the fourth (vpp-db)

---

Below you can see an ifconfig output of this container. NICs seem to connect to each network in an arbitrary way every time I docker-compose down | docker-compose up...

eth0  Link encap:Ethernet  HWaddr 02:42:ac:15:00:03  
      inet addr:172.21.0.3  Bcast:0.0.0.0  Mask:255.255.0.0

eth1  Link encap:Ethernet  HWaddr 02:42:ac:17:00:03  
      inet addr:172.23.0.3  Bcast:0.0.0.0  Mask:255.255.0.0

eth2  Link encap:Ethernet  HWaddr 02:42:ac:14:00:02  
      inet addr:172.20.0.2  Bcast:0.0.0.0  Mask:255.255.0.0

eth3  Link encap:Ethernet  HWaddr 02:42:ac:16:00:03  
      inet addr:172.22.0.3  Bcast:0.0.0.0  Mask:255.255.0.0

Edit:

Further reading, github issues:

• The order in which networks are assigned to interfaces does not match the order they are listed in compose file #4645
• 17.06 L4 ingress network not accessable #34003
• Control interface name in docker multiple networking #25181

Toughminded

Can you show us the corresponding services entry in your docker-compose.yml? – Postprandial

I added some part of my .yml file. – Toughminded

This is very similar to what Im attempting except that I wish to edit the networks in host, and Im trying to run a scrip on compose up that runs on host. Question is : #43209479 – Waksman

Answer 1

This isn't an answer, but it is a confirmation of the behavior you've reported along with complete reproducers that could be used to file a bug report.

I am able to reproduce your behavior. I think it's a bug in docker-compose, since the networks key on your service is an ordered list. I would expect networks to be assigned to interfaces in the order in which they are listed in the file.

Using this docker-compose.yml (in a directory named nwtest):

version: "2"

services:
  server:
    image: alpine
    command: sleep 999
    networks:
      - nw0
      - nw1
      - nw2
      - nw3

networks:
  nw0:
  nw1:
  nw2:
  nw3:

And this shell script:

#!/bin/sh

docker-compose up -d

for nw in 0 1 2 3; do
    nw_cidr=$(docker network inspect -f '{{ (index .IPAM.Config 0).Subnet }}' \
        nwtest_nw${nw})
    if_cidr=$(docker exec -it nwtest_server_1 ip addr show eth${nw} |
        awk '$1 == "inet" {print $2}')

    nw_net=$(ipcalc -n $nw_cidr | cut -f2 -d=)
    if_net=$(ipcalc -n $if_cidr | cut -f2 -d=)

    echo "nw${nw} $nw_net eth${nw} ${if_net}"

    if [ "$if_net" != "$nw_net" ]; then
        echo "MISMATCH: nw${nw} = $nw_net, eth${nw} = $if_net" >&2
    fi
done

docker-compose stop

You can quickly verify the problem:

$ sh runtest.sh 
Starting nwtest_server_1
nw0 192.168.32.0 eth0 192.168.32.0
nw1 192.168.48.0 eth1 192.168.48.0
nw2 192.168.64.0 eth2 192.168.80.0
MISMATCH: nw2 = 192.168.64.0, eth2 = 192.168.80.0
nw3 192.168.80.0 eth3 192.168.64.0
MISMATCH: nw3 = 192.168.80.0, eth3 = 192.168.64.0
Stopping nwtest_server_1 ... 

Furthermore, this problem seems to be specific to docker-compose; if you create a Docker container with docker run and then attach multiple networks, they are always assigned to interfaces sequentially as you would expect. The test script for that is:

#!/bin/sh

docker rm -f nwtest_server_1
docker run -d --name nwtest_server_1 --network nwtest_nw0 \
    alpine sleep 999

for nw in 1 2 3; do
    docker network connect nwtest_nw${nw} nwtest_server_1
done

for nw in 0 1 2 3; do
    nw_cidr=$(docker network inspect -f '{{ (index .IPAM.Config 0).Subnet }}' \
        nwtest_nw${nw})
    if_cidr=$(docker exec -it nwtest_server_1 ip addr show eth${nw} |
        awk '$1 == "inet" {print $2}')

    nw_net=$(ipcalc -n $nw_cidr | cut -f2 -d=)
    if_net=$(ipcalc -n $if_cidr | cut -f2 -d=)

    echo "nw${nw} $nw_net eth${nw} ${if_net}"

    if [ "$if_net" != "$nw_net" ]; then
        echo "MISMATCH: nw${nw} = $nw_net, eth${nw} = $if_net" >&2
    fi
done

docker rm -f nwtest_server_1

Postprandial

I've opened issue 4645 on GitHub. Let's see if that goes anywhere! – Postprandial

Thank you larsks! – Toughminded

Answer 2

You can use the priority attribute in Docker compose reference to specify the order which compose attach the networks to your container.

https://docs.docker.com/compose/compose-file/compose-file-v2/#priority

...

services:
  foo:
    image: foo:latest
    restart: always
    networks:
      network1:
        priority: 1000 # eth0
      network2:
        priority: 900 # eth1
      network3: # Default priority is 0, eth2

...

Macgregor

Thanks! I'll check it out 👍 – Toughminded

Waiting on github.com/docker/cli/issues/1372 for network priority support in v3 – Tanguay

Answer 3

You can try to mount config files

i.e. for RH based images:

/etc/sysconfig/network-scripts/ifcfg-eth0

So, your yml file would look like below:

# part of docker-compose.yaml

services:
  my_tool:
    build: ./my_tool
    image: tgogos/my_tool
    container_name: my_tool_docker_comp
    hostname: my_tool
    privileged: true
    links:
      - my_db
    networks:
      - vpp-nw
      - vpp-client
      - vpp-server
      - vpp-db
    volumes:
     - ./conf/eth0:/etc/sysconfig/network-scripts/ifcfg-eth0
     - ./conf/eth1:/etc/sysconfig/network-scripts/ifcfg-eth1
     - ./conf/eth2:/etc/sysconfig/network-scripts/ifcfg-eth2
     - ./conf/eth3:/etc/sysconfig/network-scripts/ifcfg-eth3
    ports:
      - "8888:8888"

Monroe

This relates to my question. #43209479 So if I run the container in privileged mode, am I able to change the current network configurations on the host. How can I make sure they take effect on the host, as if I were to make a call on shell on the host for instance: ifconfig eth0 10.0.0.100 netmask 255.255.255.0 or setting the gateway route add default gw 10.0.0.1 eth0 – Waksman

Sorry for the question spamming, If I were to share a volume with the script, not mapping it to a container path, For instance: ./path_to_script.sh in privileged mode, would it run on Host, or in the container? – Waksman