Not able to connect to wurstmeister/kafka

Question

I am running wurstmeister/kafka in a ubuntu 14.04 LTS machine. Kafka container is running fine. However I am not able to connect my microservice to this container.

Kafka docker-compose.yml:

version: '2'
services:
  zookeeper:
    image: user1/zookeeper:3.4.9
    ports:
      - "2181:2181"
  kafka:
    image: my-kafka
    ports:
      - "9092:9092"
    hostname: kafka-01
    environment:
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ADVERTISED_HOST_NAME: "kafka-01"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "topic1:1:1,topic2:1:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

docker-compose for microservice(myapp):

version: '2'

services:
  myapp:
    network_mode: 'bridge'
    extends:
      file: myapp.base.yml
      service: myapp
    links:
      - kafka

  zookeeper:
    network_mode: 'bridge'
    extends:
      file: zookeeper.yml
      service: zookeeper

  kafka:
    network_mode: 'bridge'
    extends:
      file: kafka.yml
      service: kafka
    links:
      - zookeeper

I have linked kafka in an env:

kafkaHost=kafka-01:9092

Please let me know what am I doing wrong or if more info is required. Thanks in advance.

Veracruz

Answer 1

The hostname should be what you specified in "links", ie. "kafka". As @dnephin said the hostname is not needed. I think what you want is actually KAFKA_ADVERTISED_HOST_NAME environment variable.

What version of docker do you have? I am using docker 1.12.3 on Mac OS with no issues.

You might also want to try the Confluent docker images:

• https://hub.docker.com/r/confluentinc/cp-kafka/
• https://hub.docker.com/r/confluentinc/cp-zookeeper/

Here's my sample docker compose file:

version: "2"
services:
  web:
    image: nginx:latest
    links: 
      - kafka
  zookeeper:
    extends:
      file: kafka-services.yml
      service: zookeeper
  kafka:
    extends:
      file: kafka-services.yml
      service: kafka
    depends_on:
      - zookeeper

My base kafka services compose file (kafka-services.yml):

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    ports:
      - "32181:32181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    ports:
     - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092

Olette

I used confluentinc kafka and zk images and updated my compose files as per your suggestion. Now, getting this exception: [2016-11-02 06:54:58,267] WARN [Controller-1-to-broker-1-send-thread], Controller 1's connection to broker kafka:29092 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread) kafka_1 | java.io.IOException: Connection to kafka:29092 (id: 1 rack: null) failed – Veracruz

@falcon01: what's your docker version? – Olette

I $ docker-compose version docker-compose version 1.9.0-rc3, build fcd38d3 docker-py version: 1.10.6 CPython version: 2.7.9 OpenSSL version: OpenSSL 1.0.1t 3 May 2016 – Olette

here's my version – Olette

$ docker-compose version docker-compose version 1.9.0-rc3, build fcd38d3 docker-py version: 1.10.6 CPython version: 2.7.9 OpenSSL version: OpenSSL 1.0.1t 3 May 2016 – Olette

docker-compose version 1.7.1, build 0a9ab35 docker-py version: 1.8.1 CPython version: 2.7.9 OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013 – Veracruz

@falcon01 I am not able to reproduce your error with the above compose file on ubuntu trusty. – Olette

my docker-compose version: docker-compose version 1.8.1, build 878cff1 docker-py version: 1.10.3 CPython version: 2.7.9 OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013 – Olette

Answer 2

The hostname field does not set the public hostname, it sets the internal container hostname. It's generally not that useful.

Hostnames are automatically provided by docker networking, use kafka as the hostname.

Peugia

tried both ways. removed hostname from and then set it as kafka. I am getting: o.a.kafka.common.network.Selector Selector.java:276 - Error in I/O with kafka/172.17.42.10 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.kafka.common.network.Selector.poll(Selector.java:238) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) – Veracruz

You might also have to retry the connection, as it's not necessarily available immediately. docs.docker.com/compose/startup-order – Peugia

Answer 3

It looks like this is all wurstmeister images. For kafka there is bitnami/kafak and zookeeper there is the official zookeeper image that can be used to replace for now.

Swordsman