Unable to produce to Kafka topic that is running on WSL 2 from Windows

Question

I am running the latest Kafka on Ubuntu WSL2 successfully. I can start zookeeper, kafka server, create topics, console produce and console consume just fine from within the Ubuntu that I have running on the WSL. However, when I go into my Intellij on Windows and create a simple Java Producer it does not seem to be able to connect to the broker

Versions & Hostname

    Java version: 1.8
    Kafka Version: 2.6
    hostname (from Ubuntu): KDAAPPDEV04
    hostname (from Powershell): KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getHostName() = KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getCanonicalHostName() = KDAAPPDEV04
    netstat from CMD:
        TCP    [::1]:9092             [::]:0                 LISTENING

server.properties I found this settings on another SO answer but these did not work for me.

advertised.listeners=PLAINTEXT://127.0.0.1:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

then tried (and restarted zookeeper and kafka)

advertised.listeners=PLAINTEXT://KDAAPPDEV04:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

Producer

I run this producer with three different values: hostname, localhost and 127.0.0.1 but it never connects to the broker

    public class ProducerDemo{

    private static Logger logger = LoggerFactory.getLogger(ProducerDemo.class);

    public static void main(String[] args) throws UnknownHostException{

        System.out.println(InetAddress.getLocalHost().getHostName());
        System.out.println(InetAddress.getLocalHost().getCanonicalHostName());

        String bootstrapServers = "127.0.0.1:9092";
//        String bootstrapServers = "localhost:9092";
//        String bootstrapServers = "KDAAPPDEV04:9092";

        //create Producer properties
        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServers);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());

        //create the producer
        KafkaProducer<String,String> producer = new KafkaProducer<String, String>(properties);

        //create a producer record
        ProducerRecord<String,String> record = new ProducerRecord<String, String>("first-topic","hola mundo");

        //send data
        producer.send(record);

        //flush + close
        producer.flush();
        producer.close();
    }
}

Error

[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.6.0
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 62abe01bee039651
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1601666175706
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (KDAAPPDEV04/my-ipconfig-address-here:9092) could not be established. Broker may not be available.

Simplicity

Answer 1

Had this same issue. The root cause seems to be that WSL2 is broken with regards to IPv6 and localhost (See: https://github.com/microsoft/WSL/issues/4851)

The only fix I found that doesn't involve changing configs every time you reboot (per the "172.*" suggestion above) is to use the IPv6 loopback address ::1 in both the Kafka server config running in Linux and the Java client in Windows.

In server.properties I have this:

listeners=PLAINTEXT://[::1]:9092

And likewise in my Java client bootstrap server config I use

"[::1]:9092"

Nonstandard

This was a great solution. up voted. Thanks! – Hern

Thank you. A light note, inside the WSL2, we'll need to change the bootstrap server declaration in the CLI as well. E.g: kafka-console-consumer.sh --bootstrap-server [::1]:9092 --topic my_topic – Dealt

I am running the confluent platform in wsl - kafka runs after changing this, but the connect service does not run even after changing the bootstrap server props in connect related services – Forsook

It should be noted however that the example applications (e. g. kafka.apache.org/31/documentation/streams/quickstart) don't work out of the box, as their code still uses localhost:9092 as host connection – Passacaglia

Answer 2

I had the exact problem you are having and I resolved it as follows:

1. I ran the following command in my WSL2 Ubuntu shell: ip addr | grep "eth0" I made note of the ip address against the inet property, for example, 172.27.10.68
2. In my Kafka server.properties I replaced the listeners property value as follows: listeners=PLAINTEXT://172.27.10.68:9092 I commented out the advertised.listeners property. But you can alternatively assign the ip in question to this property, and have the listeners property set to 0.0.0.0. But I assume you are using the Kafka installation for testing/learning purposes, so I would keep it simple.
3. I made no change to the Zookeeper's default ip:port
4. I am using the Schema Registry, so I modified the Kafka bootstrap property as follows: kafkastore.bootstrap.servers=PLAINTEXT://172.27.10.68:9092 I made no change to the default schema registry listener listeners=http://0.0.0.0:8081
5. I used the same ip (as listed above) in my IntelliJ Kafka Producer. It then happily connected to my Kafka broker in WSL2.

More information on WSL2 networking can be found at https://learn.microsoft.com/en-us/windows/wsl/compare-versions .

The only problem with this setup is that every time you shutdown or restart your Windows machine, or close your Ubuntu terminal, the ip address for eth0 changes. And this results in redoing steps 2, 4 and 5. I am sure there is a better way, but everything I tried failed, except for this.

Runnel

It works. But I had an issue with my connection even after this changes. Then I looked on ubuntu kafka server.log and there was an error 'ERROR Processor got uncaught exception. (kafka.network.Processor)', and I've found that the issue also can be caused by kafka-clients version missmatch in your maven dependecies. I'am also recommend check this – Tatman

Answer 3

Stop Kafka and Zookeeper, then

Disable IPv6 on WSL2:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Start Kafka, and you're good to go!

Wong

This workaround seems best, since it's a one-time fix applied to the server and doesn't require changing client config. Also keeps the fix confined to WSL itself, which is where the bug supposedly is. – Aidaaidan

@Stephane Maarek , I tried this but I am getting sysctl: cannot stat /proc/sys/net/ipv6/conf/all/disable_ipv6: No such file or directory – Anticipate

Answer 4

WSL2 runs on hypervisor and you need port proxy to connect Kafka Broker running on WSL2.

Step 1 . Check you WSL2 IP using following command and copy inet value

$ ifconfig
inet 172.X.X.X 

Step 2. Open cmd with Admin permsissions

   netsh interface portproxy add v4tov4 listenport=9092 listenaddress=0.0.0.0 connectport=9092 connectaddress=172.X.X.X

You should be able to connect now

Note : WSL2 IP changes everytime you restart machine

Icebreaker

Answer 5

I am able to find a work around . Thanks to Goose's comments

1. I ran the following command in my WSL2 Ubuntu shell: ip addr
2. Then ip address against the inet property global eth0 . for example, inet 172.20.XXX.XXX/20 .... scope global eth0
3. I replaced all localhost with this IP address in the docker-compose.yml
4. I replaced the localhost with this IP address in springboot yml or properties file.
5. My Kafka producer and consumer able to connect to the Kafka running in Ubunti - WSL 2 from Windows

Scutch

Answer 6

I got this problem when running a kafka producer in IntelliJ and a consumer in ubuntu terminal while on WSL2.

First, stop Kafka and Zookeeper. Then run these commands on WSL2, one by one:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

After that, in the kakfa folder, go to config/server.properties and edit the file to add the line:

listeners=PLAINTEXT://localhost:9092

When these commands have succeeded relaunch zookeeper and kafka.

https://www.conduktor.io/kafka/kafka-fundamentals

Hanging

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Aldis

Answer 7

This is not the optimal solution, but you will be able to connect if you run your producer in Ubuntu/WSL. This means if you are using a Windows IDE, writing the code, switching to Ubuntu and using a command line compiler and running the producer. See this post Error connecting to kafka server via IDE in WSL2

Bonina

Answer 8

1. Edit the file etc/sysctl.conf and add following lines in it.

   net.ipv6.conf.all.disable_ipv6=1

   net.ipv6.conf.default.disable_ipv6=1

2. Replace listeners=PLAINTEXT://:9092 with listeners=PLAINTEXT://localhost:9092 in your server.properties.

3. Update the sysctl config by using the following command. (Everytime you restart your machine this command needs to be run to update the configuration)

   sudo sysctl -p

Machismo

I was strugguling and tried lot of option. Finally change server.properties as below. listeners=PLAINTEXT://:9092, advertised.listeners=PLAINTEXT://172.25.205.68:9092 please make note 172.25.205.68 need to be replace with your wsl2 address. and make chanes in java code like advertised.listeners=PLAINTEXT://172.25.205.68:9092 . That will work. – Fernandofernas

Answer 9

Step 1 Disable IPv6 on WSL2:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Step 2:

In \config\server.properties under Kafka folder Replace listeners=PLAINTEXT://:9092 with listeners=PLAINTEXT://localhost:9092 in your server.properties.

Step 3: In WLS2 Ubuntu,

sudo sysctl -p

After making above changes it worked for me in my local

Sena