My kafka docker container cannot connect to my zookeeper docker container

Question

I want to use both confluent/kafka and confluent/zookeeper and run them on a single Ubuntu server.

I'm using the following configurations:

docker run -e ZOOKEEPER_CLIENT_PORT=2181 --name zookeeper confluent/zookeeper

docker run --name kafka -e KAFKA_ADVERTISED_HOST_NAME=kafka -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_CREATE_TOPICS=testtopic:1:1 confluent/kafka

However this results in: Unable to connect to zookeeper:2181

I have other containers that I'd like to connect to, how can I access zookeeper via zookeeper:2181 and kafka via kafka:9092 ?

Panlogism

Both of these containers are deprecated. Use confluentinc/kafka and Zookeeper . Also, please use Docker compose and read this blog rmoff.net/2018/08/02/kafka-listeners-explained – Theresa

Thanks @cricket_007 - I was using a docker compose, but the images started failing (it worked up until last week), so I've been running the images as per above to help to debug the failing container. I will try the commands above, with confluentinc/kafka and confluentinc/zookeeper – Panlogism

Answer 1

There are multiple ways to do this. But before we look into it there are two problems in your approach that you need to understand

• zookeper host is not reachable when you use docker run as each of the containers is running in a different network isolation
• kafka may start and try to connect to zookeeper but zookeeper is not ready yet

Solving the network issue

You can do a lot of things to fix things

use --net=host to run both on the host network

use docker network create <name> and then use --net=<name> while launching both the containers

Or you can run your kafka container on the zookeeper containers network.

use --net=container:zookeeper when launching kafka container. This will make sure zookeeper host is accessible. This is not recommended as such, until unless you have some strong reason to do so. Because as soon as zookeeper container goes down, so will be the network of your kafka container. But for the sake of understanding, I have put this option here

Solving the startup race issue

Either you can keep a gap between starting zookeeper and kafka, to make sure that when kafka starts zookeeper is up and running

Another option is to use --restart=on-failure flag with docker run. This will make sure the container is restarted on failure and will try to reconnect to zookeeper and hopefully that time zookeeper will be up.

Instead of using docker run I would always prefer the docker-compose to get such linked containers to be run. You can do that by creating a simple docker-compose.yml file and then running it with docker-compsoe up

version: "3.4"
services:
  zookeeper:
    image: confluent/zookeeper
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181
  kafka:
    image: confluent/kafka
    environment:
      - KAFKA_ADVERTISED_HOST_NAME=kafka
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_CREATE_TOPICS=testtopic:1:1
    depends_on:
      - zookeeper
    restart: on-failure

Saxtuba

docker run -e ZOOKEEPER_CLIENT_PORT=2181 --name zookeeper confluent/zookeeper followed by docker run --name kafka -e KAFKA_ADVERTISED_HOST_NAME=kafka -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_CREATE_TOPICS=testtopic:1:1 --net=container:zookeeper confluent/kafka did not work. And showed "Unable to connect to zookeeper:2181" – Panlogism

Did you do it after a delay? – Saxtuba

Yes, I executed the first command, it showed : "INFO binding to port 0.0.0.0/0.0.0.0:2181", then waited 2 minutes, checked it was still running, then executed the second command. – Panlogism

Try with localhost:2181 then – Saxtuba

Thanks that works! I would like this to work with zookeeper though. So perhaps there is another environment variable for advertised_name with zookeeper. – Panlogism

Best is to use docker-compose instead of just docker – Saxtuba

Answer 2

I'm running on Mac though, this is working fine. since 'host' networking not working in mac i just create a network called kafka_net and put the containers there.

version: "3.4"
services:
  zookeeper:
    image: confluent/zookeeper
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181
    networks:
      - kafka_net
  kafka:
    image: confluent/kafka
    environment:
      - KAFKA_ADVERTISED_HOST_NAME=kafka
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
    depends_on:
      - zookeeper
    networks:
      - kafka_net
    restart: on-failure
networks:
  kafka_net:
    driver: "bridge"

To make sure all working:

Log into the zookeeper container then

zookeeper-shell localhost:2181 => You should see something like 'Welcome to ZooKeeper!' after all the big chunk of text

Log into kafka container then

kafka-topics --zookeeper zookeeper:2181 --list # empty list
kafka-topics --zookeeper zookeeper:2181 --create --topic first_topic --replication-factor 1 --partitions 1
kafka-topics --zookeeper zookeeper:2181 --list # you will see the first_topic
kafka-console-producer --broker-list localhost:9092 --topic first_topic # type some text and ctrl + c
kafka-console-consumer --bootstrap-server localhost:9092 --zookeeper zookeeper:2181 --topic first_topic --from-beginning # you will see the stuff you typed first_topic

If still giving problems have a look in the official examples. https://github.com/confluentinc/cp-docker-images/tree/5.2.2-post/examples and still giving issues post it, will give a hand.

Exist

much better answer – Unctuous

Answer 3

Docker start containers in isolated network, called default bridge unless you specify network explicitly.

You can succeed in different ways, here are 2 easiest:

1. Put containers into same user-defined bridge network

   # create net
   docker network create foo
   docker run --network=foo -e ZOOKEEPER_CLIENT_PORT=2181 --name zookeeper confluent/zookeeper
   docker run --network=foo --name kafka -e KAFKA_ADVERTISED_HOST_NAME=kafka -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_CREATE_TOPICS=testtopic:1:1 confluent/kafka
   

2. Expose ports and connect through localhost

   docker run -p 2181:2181 -e ZOOKEEPER_CLIENT_PORT=2181 --name zookeeper confluent/zookeeper
   docker run --name kafka -e KAFKA_ADVERTISED_HOST_NAME=kafka -e KAFKA_ZOOKEEPER_CONNECT=host.docker.internal:2181 -e KAFKA_CREATE_TOPICS=testtopic:1:1 confluent/kafka
   

Note: in second approach you should use host.docker.internal as a host name and expose (publish) port 2181 for first container to make it available on localhost

Myocardium

I tried the first option, and when running the second line it throws "org.I0Itec.zkclient.exception.ZkException: Unable to connect to zookeeper:2181" – Panlogism

I tried the second option, and I also get Unable to connect to zookeeper:2181 – Panlogism

KAFKA_CREATE_TOPICS isn't valid for these containers, also, you don't need the docker host DNS name here – Theresa

Answer 4

 docker network create kafka-zookeeper

Wait until Network is Created

 docker run -it -d --network=kafka-zookeeper --name zookeeper zookeeper

Wait until ZooKeeper is up and Running

 docker run -it -d --network=kafka-zookeeper --name kafka -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 --restart=on-failure -e ALLOW_PLAINTEXT_LISTENER=yes -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092 -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 bitnami/kafka

Kafka Should be connecting fine.

These are running in -d detached mode, so you need to go to Docker Desktop to view the logs for each container.

Graphemics