Interact with kafka docker container from outside of docker host [duplicate]

Question

I have built a kafka docker container and orchestrate it using docker-compose.

Calling docker ps I get the following putput:

CONTAINER ID        IMAGE                          COMMAND                CREATED             STATUS              PORTS                                         NAMES
    5bde6f76246e        hieutrtr/docker-kafka:0.0.1    "/start.sh"            About an hour ago   Up About an hour    7203/tcp, 0.0.0.0:32884->9092/tcp             dockerkafka_kafka_3
    be354f1b8cc0        hieutrtr/docker-ubuntu:devel   "/usr/bin/supervisor   About an hour ago   Up About an hour    22/tcp                                        producer1
    50d3203af90e        hieutrtr/docker-kafka:0.0.1    "/start.sh"            About an hour ago   Up About an hour    7203/tcp, 0.0.0.0:32883->9092/tcp             dockerkafka_kafka_2
    61b285f39615        hieutrtr/docker-kafka:0.0.1    "/start.sh"            2 hours ago         Up 2 hours          7203/tcp, 0.0.0.0:32882->9092/tcp             dockerkafka_kafka_1
    20c9c5ccec05        jplock/zookeeper:3.4.6         "/opt/zookeeper/bin/   2 hours ago         Up 2 hours          2888/tcp, 3888/tcp, 0.0.0.0:32881->2181/tcp   dockerkafka_zookeeper_1

I can run a producer and a consumer from inside the docker container, but it is not working from outside the docker network.

For example:

I run a kafka producer on my localhost and the following error appears:

$ kafka_2.9.1-0.8.2.1: bin/kafka-console-producer.sh --topic test --broker-list $DOCKER_HOST:32884
[2015-08-31 06:55:15,450] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
to
[2015-08-31 06:55:20,214] WARN Failed to send producer request with correlation id 2 to broker 1 with data for partitions [test,0] (kafka.producer.async.DefaultEventHandler)
java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
    at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:103)
    at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)
    at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)

This is my kafka docker example on github that includes the mentioned problem.

So, is anyone experiencing the same problems and can help me in any way?

Additional info:

(Just fork from ches/kafka and modify something for docker-compose) :

Falstaffian

Answer 1

In the Kafka server properties you need to set advertised.listeners to the ip/port of your running container and then it should work.

Thelmathem

i think it's not the problem. Because i still create topic from my localhost. Is there anything about java.rmi.... ? – Curson

This is what worked for me for very similar symptoms. I could create topics from localhost or other containers but not publish any messages until I set the advertised.host.name – Thelmathem

what should we put as advertised.host.name? Facing the same issue – Wilbanks

advertised.host.name is now deprecated and I had to use advertised.listener. – Donitadonjon

Answer 2

Here are my two cents, since I had a hard time figuring this one out.

My $KAFKA_HOME/config/server.properties contains the following:

listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT

advertised.listeners=INSIDE://${container_ip}:9092,OUTSIDE://${outside_host_ip}:29092

listeners=INSIDE://:9092,OUTSIDE://:29092

inter.broker.listener.name=INSIDE

This is creating two connections, one to be used inside docker and another to be used outside. You have to choose a new port for the latter, in my case 29092, make sure this port is exposed and mapped by docker.

I was not yet able to figure out a solution without the ${outside_host_ip} in the environment, hence I'm providing the host machine's ip as an env var.

Test:

1. Enter the Kafka container and create a topic: ./kafka-topics.sh -zookeeper zookeeper:2181 --create --topic dummytopic --partitions 1 --replication-factor 1
2. From outside the Kafka container do: ./kafka-console-producer.sh --broker-list 0.0.0.0:29092 --topic dummytopic and input a message

I hope this helps others

Unerring

Noted that the ports inside and outside have to be different. I don't understand why that's the case. Does anyone have an idea why Kafka throws an exception when those ports are the same? You'd think that the ports inside the Docker network can be whatever, and that shouldn't have any impact on the ports outside the Docker network. Kafka throws an exception from some Java argument parser if the ports are the same. – Nannienanning

Answer 3

You need to put the name of the host machine where the docker instance was deployed. Also you need to map ports from docker host machine(public) to the docker container instance(private).

Earthshaker

Also make sure you dont set any of the related propeties in server.properties – Earthshaker

Sorry hit enter too early. – Earthshaker

# The id of the broker. This must be set to a unique integer for each broker. broker.id=0 #listeners=PLAINTEXT://localhost:9092 #advertised.listeners=PLAINTEXT://localhost:9092 #advertised.host.name=localhost #advertised.port=9092 – Earthshaker

Pardon one of my first time usage. The values set in the server.properties take precedence and in my experience, the port is taken by default to 9092. I run dev and qa docker Kafka in the same Kafka. Did not set the port anywhere, but using the exposed ports from docker container as 9092. Mapping these ports to public/docker host port to 9092 and 9192 for dev and qa respectively. – Earthshaker

Answer 4

TL;DR Expose port 9092 on host and map it to 9092 container port to access kafka brokers outside container. See docker-compose documentation for details.

I think the problem is that you don't expose port 9092 outside container. According to your docker ps listing your 9092 container port is mapped dynamically to host's port range 32882-32884. When you connect to broker configured this way you receive metadata that contains port 9092 for advertising. With this metadata producer tries to do other requests through port 9092 and fails.

Pulverize

Answer 5

For the record, another way to get my local kafka consumer communicating with remote broker inside a Docker container was to add an entry in my /etc/hosts : docker-host-ip-address docker-kafka-container-hostname

Anyway Lundahl's solution worked fine for me and seems cleaner. Even cleaner would be to set advertised.listeners=host-ip:port since advertised.host.name and advertised.port are deprecated.

Counterblast