Not able to connect to kafka server on google compute engine from local machine

Question

I am running my zookeeper and kafka server on google compute engine. Both are running on default ports(zookeeper on 2181 an kafka on 9092). Both are running on the same instance. I have opened up both the ports as well. In my server.properties I have configured

zookeeper.connect=<InternalIP>:2181
host.name=localhost

If I try to push/consume message form the same server, I am able to do so To push/consume I use

 bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
 bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

However, If try to the same from my local machine I get kafka.common.FailedToSendMessageException in producer and java.net.ConnectException: connection refuesed in case of consumer I try to push/consume via

bin/kafka-console-producer.sh --broker-list <ExternalIP>:9092 --topic topic1
bin/kafka-console-consumer.sh --zookeeper <ExternalIP>:2181 --topic topic1 --from-beginning

Please note that i am able to ping the external ip from my local system.

I have configured the below mentioned firewall rules in compute engine

Description

kafka port enabled
Network

default
Source filter

Allow from any source (0.0.0.0/0)
Allowed protocols and ports

tcp:9092

Description

zookeeper port enabled
Network

default
Source filter

Allow from any source (0.0.0.0/0)
Allowed protocols and ports

tcp:2181

Roundel

That's because zookeeper is binding with the internal IP and not with the external IP. – Salt

Both zookeeper and kafka are running on the same vm instance, hence I used the internal IP to bind. – Roundel

Answer 1

You must access to the cloud compute VM instance through SSH, then edit the kafka configuration file.

$ sudo vim /opt/bitnami/kafka/config/server.properties

Uncomment the line # advertised.listeners=PLAINTEXT://:9092 and replace with advertised.listeners=PLAINTEXT://[instance_public_id_address]:9092

As a last step restart the kafka service

sudo /opt/bitnami/ctlscript.sh restart

It's important to consider which the default IP address of the GCP compute VM is ephemeral so you must change it to static in the GCP Configuration panel of the Kafka instance, in order to avoid change the configuration file each time that the IP address changes.

Trout

this does not work for me. note: my listener is set to default. – Larentia

Answer 2

Have you configured any firewall rules? You don't mention it, so I assume not.

From https://cloud.google.com/compute/docs/networks-and-firewalls#firewalls :

"By default, all incoming traffic from outside a network is blocked and no packet is allowed into an instance without an appropriate firewall rule."

Downtown

I have configured the firewall rules. Have edited my question to reflect the same – Roundel