Cannot connect to single-node Kafka server through Docker

Question

I'm trying to connect to single-node Kafka server through Docker but I am getting the following error:

%3|1529395526.480|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused
%3|1529395526.480|ERROR|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused
%3|1529395526.480|ERROR|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down

The docker-compose.yml file contents are as follows:

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    network_mode: host
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    extra_hosts:
      - "moby:127.0.0.1"

  kafka:
    image: confluentinc/cp-kafka:latest
    network_mode: host
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: localhost:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_ADVERTISED_HOSTNAME: kafka
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    extra_hosts:
      - "moby:127.0.0.1"

  schema_registry:
    image: confluentinc/cp-schema-registry
    hostname: schema_registry
    depends_on:
      - zookeeper
      - kafka
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema_registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: '127.0.0.1:2181'

The Dockerfile contents are the following:

FROM python:2

WORKDIR /kafkaproducerapp

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "python", "./BackOffice_Producer.py" ]

What am I doing wrong?

Karen

Have you tried adding port 9092 to your kafka? – Medford

you are not mapping the port of the container to your host as you did for the schema registry, also for the host you can add it to the host file of the container so it is known in place of localhost see this example for a java app – Ader

Added the port but i'm still getting the same error. – Karen

Where did you get this compose file? You should not use localhost or 127.0.0.1 to connect one container to the others... Basically, Kafka nor the Registry can reach Zookeeper – Mortgagee

Answer 1

You need this:

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092

Otherwise the Kafka brokers will be telling anyone connecting that it can be found on localhost:9092, which is not going to work from the other containers. From your other containers use kafka:29092 as the broker host & port, as well as zookeeper:2181 for zookeeper.

From your local host machine, you can access your broker on 9092 (assuming you expose the port).

Check out this docker-compose for a full example (from this repo)

Proceeding

For anyone interested the docker-compose url changed to https://raw.githubusercontent.com/confluentinc/cp-all-in-one/master/cp-all-in-one/docker-compose.yml – Gerlach