ZeroMQ failing to communicate between two Docker containers

Question

I'm trying to set up a toy example of Docker networking with ZeroMQ in macOS, where the serverd.py sends a message to the clientd.py and the client simply displays it using PUSH/PULL. If I run them outside of the container they work fine, but I'm having trouble getting them to communicate when running within separate containers. It seems like my clientd.py cannot connect to the container name despite them being within the same bridge network. I tried replacing the hostname with the assigned ip address for serverd_dev_1, but this doesn't work either.

Here's my setup:

1. I created a new network with docker network create -d bridge mynet. Here is the output from docker network inpsect mynet:

   {
       "Name": "mynet",
       "Id": "cec7f8037c0ef173d9a9a66065bb46cb6a631fea1c0636876ccfe5a792f92412",
       "Created": "2017-08-19T09:52:44.8034344Z",
       "Scope": "local",
       "Driver": "bridge",
       "EnableIPv6": false,
       "IPAM": {
           "Driver": "default",
           "Options": {},
           "Config": [
               {
                   "Subnet": "172.18.0.0/16",
                   "Gateway": "172.18.0.1"
               }
           ]
       },
       "Internal": false,
       "Attachable": false,
       "Ingress": false,
       "ConfigFrom": {
           "Network": ""
       },
       "ConfigOnly": false,
       "Containers": {
           "5fa8dc2f8059d675dfd3dc4f2e50265be99361cd8a8f2730eb273772c0148742": {
               "Name": "serverd_dev_1",
               "EndpointID": "3a62e82b1b34d5c08f2a9f340ff93aebd65c0f3dfde70e354819befe21422d0b",
               "MacAddress": "02:42:ac:12:00:02",
               "IPv4Address": "172.18.0.2/16",
               "IPv6Address": ""
           },
           "ec1e5f8c525ca8297611e02bcd3a64198fda3a07ce8ed82c0c4298609ba0357f": {
               "Name": "clientd_dev_1",
               "EndpointID": "a8ce6f178a225cb2d39ac0009e16c39abdd2dae02a65ba5fd073b7900f059bb8",
               "MacAddress": "02:42:ac:12:00:03",
               "IPv4Address": "172.18.0.3/16",
               "IPv6Address": ""
           }
       },
       "Options": {},
       "Labels": {}
   }
   

2. I created serverd.py and clientd.py like this and put them in separate folders along with their Dockerfiles and docker-compose.yml:

serverd.py:

import zmq
import time

context = zmq.Context()
socket = context.socket(zmq.PUSH)
address = "tcp://127.0.0.1:5557"
socket.bind(address)
print("Sending to {}...".format(address))
while True:
    message = socket.send_string("Got it!")
    print("Sent message")
    time.sleep(1)

clientd.py:

import zmq

context = zmq.Context()
socket = context.socket(zmq.PULL)
address = "tcp://serverd_dev_1:5557"
socket.connect(address)
print("Listening to {}...".format(address))
while True:
    message = socket.recv_string()
    print("Client got message! {}".format(message))

I have two Dockerfiles and docker-compose.yml:

Dockerfile for serverd.py:

FROM python:3.6

RUN mkdir src
ADD serverd.py /src/
RUN pip install pyzmq
WORKDIR /src/
EXPOSE 5557

Dockerfile for clientd.py:

FROM python:3.6

RUN mkdir src
ADD clientd.py /src/
RUN pip install pyzmq
WORKDIR /src/
EXPOSE 5557

docker-compose.yml for serverd.py:

dev:
  build: .
  command: ["python", "-u", "./serverd.py"]
  net: mynet

docker-compose for clientd.py:

dev:
  build: .
  command: ["python", "-u", "./clientd.py"]
  net: mynet

3. serverd.py starts up as expected with docker-compose up:

Sending to tcp://127.0.0.1:5557...

4. clientd.py won't start up like this due to it not finding the hostname tcp://serverd_dev_1:5557.

   Attaching to countd_dev_1
   dev_1  | Traceback (most recent call last):
   dev_1  |   File "./countd.py", line 6, in <module>
   dev_1  |     socket.connect(address)
   dev_1  |   File "zmq/backend/cython/socket.pyx", line 528, in zmq.backend.cython.socket.Socket.connect (zmq/backend/cython/socket.c:5971)
   dev_1  |   File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:10014)
   dev_1  | zmq.error.ZMQError: Invalid argument
   

5. If I replace the URI tcp://serverd_dev_1:5557 with tcp://172.18.0.2:5557 it does not crash anymore, but it is simply idling without receiving any of the messages from the server. Obviously I'm doing something wrong, but I'm not entirely sure what exactly. I feel like I have been following the Docker documentation as closely as possible, and would very much be grateful if you had any ideas.

Simonesimoneau

My suspect is a docker-isolation of devices. Would you mind to set the target addresses in an IPv4-absolute-format on the .bind()-side, as address = "tcp://172.18.0.2:5557" not to rely on any DNS-resolution and using it also as the respective .connect()-target on the client-side? EXPOSE seems just to mediate port-management between host 0/S resources management and docker-abstraction used for isolation, so the internal ( inside-docker-container ) code ought use the properly setup <transport-class>:\\address:port# on their own view of the world ( from an insider perspective ). – Pennypennyaliner

Answer 1

Your primary problem is that you have configured your server with the address tcp://127.0.0.1:5557. Because it's bound to localhost (127.0.0.1), that socket isn't going to be visible to anything outside of that container. So the first thing you need to fix is the server bind address. Consider:

address = "tcp://0.0.0.0:5557"

A second problem is that you're using the name serverd_dev_1 in the client, but it's not clear this would actually be the name of your serverd container(that would depend on the directory names in use when you run docker-compose up).

Naming is easier to manage with a single docker-compose.yaml file. For example, I set things up like this:

version: "2"

services:
  serverd:
    build: serverd
    command: ["python", "-u", "./serverd.py"]
    environment:
      SERVER_LISTEN_URI: tcp://0.0.0.0:5557

  clientd:
    build: clientd
    command: ["python", "-u", "./clientd.py"]
    environment:
      SERVER_CONNECT_URI: tcp://serverd:5557

This will launch both containers in a dedicated network (because this is what docker-compose does by default), so you don't need to explicitly create or reference mynet.

As you can probably infer from the above, I modified your code to get the ZMQ uri from an environment variable, because this made it easier to experiment. You can find the above docker-compose.yaml and the modified code at:

• https://github.com/larsks/so-example-jimmyc

Update

In case you really want/need to have two separate docker-compose.yaml files, I've updated the example to include per-service files. These examples use the alias option to provide a name by which the client can contact the server, regardless of your local directory layout:

version: "2"

services:
  serverd:
    build: .
    command: ["python", "-u", "./serverd.py"]
    environment:
      SERVER_LISTEN_URI: tcp://0.0.0.0:5557
    networks:
      mynet:
        aliases:
          - serverd

networks:
  mynet:
    external: True

This configuration requires that you create mynet before bringing up the containers.

Pane

With all due respect, the O/P has posted the code-under-test right from the begining ( item 2 above, both the serverd.py and clientd.py ). The code ( with having properly fixed the <transport-class>, DNS / hostname resolution and IPv4-address:port# ) works, so the hypothesis about a bug inside the ZeroMQ-part fails to be correct. – Pennypennyaliner

@Pane Thank you for such a thorough answer! I was confused on two parts: I assumed since examples using port forwarding to the host machine used localhost, so would using docker network. Secondly, I thought Containers ->Name in docker network inspect mynet was the name to use as host name. Your suggestion to combine the docker-compose.yml made everything a lot better as well. Thanks for taking your time! – Simonesimoneau

@Pennypennyaliner it looks like the person complaining about bugs has pulled his comments, so I'm going to remove mine as well since they're not really relevant to the answer... – Pane

@Pane Amazing effort! It truly helped me on many fronts as a Docker newbie. It is people like you that make stackoverflow such an amazing resource. – Investigation