Force kafka to connect brokers through IPs, not via hostnames

Question

We have following kafka-ssh-tunneling setup.

ssh -N $JUMPHOST -L 2181:w.x.y.z:2181 -L 9092:a.b.c.d:9092 -L 9091:e.e.f.f:9092

• broker IP is a.b.c.d , There is local lo0 device alias with same IP address
• zookeper IP is w.x.y.z , There is local device alias with same IP address
• kafkahost "entry" host is e.e.f.f

Our planned use case is kafkacat -C -b localhost:9091 -t <topic>

Problem: Connecting to kafka host/ zookepers works fine, however kafka clients ( e.g. kafkacat ) are accessing brokers by their hostname, ip-a.b.c.d.eu-central-1.compute.internal , not by their IP's.

To counteract, I've added entry to /etc/hosts

a.b.c.d ip-a.b.c.d.eu-central-1.compute.internal

Still doesn't work, although pinging to that hostname is successful.

Nslookup gives

Non-authoritative answer:
Name:   ip-a.b.c.d.eu-central-1.compute.internal
Address: a.b.c.d
** server can't find ip-a.b.c.d.eu-central-1.compute.internal: NXDOMAIN

Question: Is there a way telling kafka to connect brokers through IP's and not via hostnames? If not, will starting local dns server might resolve the issue?

Classis

Answer 1

What is happening here is:

1. The broker receives a petition from the client and returns him ip-a.b.c.d.eu-central-1.compute.internal, as this is the host name of the broker and the default value for listeners. (this is the key)

2. Your client tries to send data to the broker using the metadata it was given. But as it can't resolve ip-a.b.c.d.eu-central-1.compute.internal, it fails without even reaching the cluster, caused by a networking issue out of Kafka's scope.

   If you set the value on /etc/hosts, you will fix the address resolution problem; The client will now be able to reach the cluster, solving the previous networking issue.

   The following step involves Kafka replying with the 448_in_your_face error code (the exact code name may differ). Your petition fails again, now on cluster-side: your client is asking for a broker called/referenced a.b.c.d, but there is no registered listener with that name, as its identifier is still ip-a.b.c.d.eu-central-1.compute.internal.

---

The key here is within the advertised.listeners property, located in the server.properties configuration file. In order your clients to be able to connect, modify that property, directly setting the ip there, or a resolvable dns (using IP for this example):

advertised.listeners=PLAINTEXT://a.b.c.d:9092

Now on client side, just use the IP in order to connect with the broker:

bootstrap.servers = a.b.c.d:9092

When the petition from the client is received, kafka will recognize the content of bootstrap.servers as one of its registered listeners, hence accepting the connection.

Bani

Thanks for dig-in to my complex problem. I will ask our devOps team to see if they can change it, however I've found another work-around as solution, I will post it as just in case someone will need it. – Classis

See this answer here for more on how advertised.listeners works: https://mcmap.net/q/233954/-kafka-server-configuration-listeners-vs-advertised-listeners – Millipede

Answer 2

Found workaround. Posting here If anyone might face my problem.

Steps are following:

• Create dummy aliases for all host you are planning to use, sudo ip add a dev lo $ip
• These aliases should NOT have the same broker/zookeper IPs, BUT 127.0.j.k format
• Add ip-<>.<>.<>.<>.eu-central-1.compute.internal < -- > 127.0.[].[] mapping to /etc/hosts
• Create tunnel via SSH, taking account relation of broker/zookeper's IPs and your local (aliased) IPs

ssh -N $JUMPHOST -L 2181:<localIP>:<remoteIP>:2181 -L 9092:<localIP>:<remoteIP>:9092 ...

then you can consume messages via kafkacat -C -b 127.0.[].[]:9092 -t <topic>

Classis

Why not edit the advertised.listeners to return the IP addresses? – Guanaco

@Guanaco yes, that was mentioned before. I will ask our DevOps to see if they can do it, but usually they are very conservative to any changes : ) – Classis

Otherwise, yeah, compute.internal DNS names would need to be mapped externally via an ELB, for example – Guanaco