Why can't I connect to Kafka/Zookeeper? (In a Docker)

Question

I'm running Kafka (0.10.0.0) in a Docker on a Mac (w/docker-machine). I derived my Dockerfile from Spotify's, which means Kafka and Zookeeper run in the same image.

My instance starts cleanly and poking around inside it appears everything is normal/OK.

Docker maps ports 2181 and 9092 to high-ports 32822 and 32820 in this case. From outside my running Kafka Docker I am able to successfully telnet 192.168.99.100 32822 (where 192.168.99.100 is the IP of my docker-machine). From there I can issue a zookeeper command and get expected output.

It all seems so encouraging, but... I then try this code:

val numPartitions = 4
val replicationFactor = 1
val topicConfig = new java.util.Properties

// zookeeper = "192.168.99.100:32822"
val zkClient = ZkUtils(zookeeper, 10000, 10000, false)
try {
  AdminUtils.createTopic(zkClient, topic, numPartitions, replicationFactor, topicConfig)
} catch {
  case k: kafka.common.TopicExistsException => // do nothing...topic exists
}
zkClient.close()

This produces this error output:

DEBUG ZkConnection - Creating new ZookKeeper instance to connect to 192.168.99.100:32822.
INFO  ZkEventThread - Starting ZkClient event thread.
INFO  ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
INFO  ZooKeeper - Client environment:host.name=172.25.42.82
INFO  ZooKeeper - Client environment:java.version=1.8.0_60
INFO  ZooKeeper - Client environment:java.vendor=Oracle Corporation
INFO  ZooKeeper - Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre
INFO  ZooKeeper - Client environment:java.class.path=/usr/local/Cellar/sbt/0.13.11/libexec/sbt-launch.jar
INFO  ZooKeeper - Client environment:java.library.path=/Users/wmy965/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
INFO  ZooKeeper - Client environment:java.io.tmpdir=/var/folders/ph/ccz4n1qs62n0bn8mqdg94gswt1jlwk/T/
INFO  ZooKeeper - Client environment:java.compiler=<NA>
INFO  ZooKeeper - Client environment:os.name=Mac OS X
INFO  ZooKeeper - Client environment:os.arch=x86_64
INFO  ZooKeeper - Client environment:os.version=10.11.5
INFO  ZooKeeper - Client environment:user.name=wmy965
INFO  ZooKeeper - Client environment:user.home=/Users/wmy965
INFO  ZooKeeper - Client environment:user.dir=/Users/wmy965/git/LateKafka
INFO  ZooKeeper - Initiating client connection, connectString=192.168.99.100:32822 sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@55397e3
DEBUG ClientCnxn - zookeeper.disableAutoWatchReset is false
DEBUG ZkClient - Awaiting connection to Zookeeper server
INFO  ZkClient - Waiting for keeper state SyncConnected
INFO  ClientCnxn - Opening socket connection to server 192.168.99.100/192.168.99.100:32822. Will not attempt to authenticate using SASL (unknown error)
WARN  ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
DEBUG ClientCnxnSocketNIO - Ignoring exception during shutdown input
java.nio.channels.ClosedChannelException
    at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:780)
    at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:399)
    at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:200)
    at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1110)
DEBUG ClientCnxnSocketNIO - Ignoring exception during shutdown output
java.nio.channels.ClosedChannelException
    at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:797)
    at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:407)
    at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:207)
    at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1110)
INFO  ClientCnxn - Opening socket connection to server 192.168.99.100/192.168.99.100:32822. Will not attempt to authenticate using SASL (unknown error)
INFO  ClientCnxn - Socket connection established to 192.168.99.100/192.168.99.100:32822, initiating session
DEBUG ClientCnxn - Session establishment request sent on 192.168.99.100/192.168.99.100:32822
INFO  ClientCnxn - Session establishment complete on server 192.168.99.100/192.168.99.100:32822, sessionid = 0x155225c51720000, negotiated timeout = 10000
DEBUG ZkClient - Received event: WatchedEvent state:SyncConnected type:None path:null
INFO  ZkClient - zookeeper state changed (SyncConnected)
DEBUG ZkClient - Leaving process event
DEBUG ZkClient - State is SyncConnected
DEBUG ClientCnxn - Reading reply sessionid:0x155225c51720000, packet:: clientPath:null serverPath:null finished:false header:: 1,8  replyHeader:: 1,1,-101  request:: '/brokers/ids,F  response:: v{}

It looks like I can't connect (presumably to zookeeper). Why not?

Snip

Ok, read somewhere about advertised.listeners in Kafka's server.properties file. Apparently this is what Kafka advertises to publishers/consumers when asked, so I think this has to be Docker-ized, meaning set to 192.168.99.100: 32822 (or whatever the mapped port for 9092 is). I've got that set but still no dice. I'm going to try setting the listeners property in the same config file and see what happens. – Snip

Changed the listeners property too but with these 2 changes it still won't connect. :-( I'm going to leave the advertised.listeners set to outside-the-Docker values, as I think this is right. Going to play with other settings for the listeners property tho--it may be something internal to Kafka and Zookeeper, which are in the same Docker. – Snip

What does your docker inspect <container-id> return? – Albatross

Answer 1

In new kafka streams, the ip of producer must have been knowing by kafka (docker). Kafka send their uuid (you can show this in /etc/hosts inside kafka docker) and espect response from this.

Summary:

Map uuid kafka docker to docker-machine in /etc/host of mac OS.

To help you, how to change etc/host file in mac:

https://www.tekrevue.com/tip/edit-hosts-file-mac-os-x/

Phlox

Adding the kafka docker internal hostname to my host file and mapping it to 127.0.0.1 fixed it for me. – Montoya

Answer 2

Cleaner would be to set advertised.listeners=host-ip:port since advertised.host.name and advertised.port are deprecated in Kafka server.properties file.

If set host-ip to 0.0.0.0 it will listen requests from anywhere. But it's insecure.

Dallis