ERROR org.apache.kafka.common.utils.KafkaThread - Uncaught exception in thread 'kafka-producer-network-thread

Question

i am trying to create producer and send some data to it from intellij idea but while running this program i got ERROR like ERROR org.apache.kafka.common.utils.KafkaThread - Uncaught exception in thread 'kafka-producer-network-thread

i have written code for this program is like

package Kafka;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class First_Producer {
    public static void main(String[] args) {


        String bootstrapserver = "127.0.0.1:9092";

        //create producer properties
        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapserver);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());

        //create the producer
        KafkaProducer<String,String> producer= new KafkaProducer<String,String>(properties);

        //create a producer record
        ProducerRecord<String, String>  record = new ProducerRecord<String,String>("first_topic","hello_world");

        //send data
        producer.send(record);
        producer.flush();
        producer.close();
    }
}

pom.xml file

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>KafkaProject</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.6.0</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.30</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
            <version>3.0.0-preview</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>
</project>

While running got error

     C:\Java\jdk1.8.0_161\bin\java.exe "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2020.1.4\lib\idea_rt.jar=7640:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2020.1.4\bin" -Dfile.encoding=UTF-8 -classpath C:\Java\jdk1.8.0_161\jre\lib\charsets.jar;C:\Java\jdk1.8.0_161\jre\lib\deploy.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\access-bridge-64.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\cldrdata.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\dnsns.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\jaccess.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\jfxrt.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\localedata.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\nashorn.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\sunec.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\sunjce_provider.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\sunmscapi.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\sunpkcs11.jar;C:\Java\jdk1.8.0_161\jre\lib\ext\zipfs.jar;C:\Java\jdk1.8.0_161\jre\lib\javaws.jar;C:\Java\jdk1.8.0_161\jre\lib\jce.jar;C:\Java\jdk1.8.0_161\jre\lib\jfr.jar;C:\Java\jdk1.8.0_161\jre\lib\jfxswt.jar;C:\Java\jdk1.8.0_161\jre\lib\jsse.jar;C:\Java\jdk1.8.0_161\jre\lib\management-agent.jar;C:\Java\jdk1.8.0_161\jre\lib\plugin.jar;C:\Java\jdk1.8.0_161\jre\lib\resources.jar;C:\Java\jdk1.8.0_161\jre\lib\rt.jar;D:\KafkaProject\target\classes;D:\KafkaProject\slf4j-simple-1.7.30.jar;C:\Users\vishal\.m2\repository\org\apache\kafka\kafka-clients\2.6.0\kafka-clients-2.6.0.jar;C:\Users\vishal\.m2\repository\com\github\luben\zstd-jni\1.4.4-7\zstd-jni-1.4.4-7.jar;C:\Users\vishal\.m2\repository\org\lz4\lz4-java\1.7.1\lz4-java-1.7.1.jar;C:\Users\vishal\.m2\repository\org\xerial\snappy\snappy-java\1.1.7.3\snappy-java-1.1.7.3.jar;C:\Users\vishal\.m2\repository\org\slf4j\slf4j-api\1.7.30\slf4j-api-1.7.30.jar;C:\Users\vishal\.m2\repository\org\slf4j\slf4j-simple\1.7.30\slf4j-simple-1.7.30.jar Kafka.First_Producer
        SLF4J: Class path contains multiple SLF4J bindings.
        SLF4J: Found binding in [jar:file:/D:/KafkaProject/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: Found binding in [jar:file:/C:/Users/vishal/.m2/repository/org/slf4j/slf4j-simple/1.7.30/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
        SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
        [main] INFO org.apache.kafka.clients.producer.ProducerConfig - ProducerConfig values: 
            acks = 1
            batch.size = 16384
            bootstrap.servers = [127.0.0.1:9092]
            buffer.memory = 33554432
            client.dns.lookup = use_all_dns_ips
            client.id = producer-1
            compression.type = none
            connections.max.idle.ms = 540000
            delivery.timeout.ms = 120000
            enable.idempotence = false
            interceptor.classes = []
            internal.auto.downgrade.txn.commit = false
            key.serializer = class org.apache.kafka.common.serialization.StringSerializer
            linger.ms = 0
            max.block.ms = 60000
            max.in.flight.requests.per.connection = 5
            max.request.size = 1048576
            metadata.max.age.ms = 300000
            metadata.max.idle.ms = 300000
            metric.reporters = []
            metrics.num.samples = 2
            metrics.recording.level = INFO
            metrics.sample.window.ms = 30000
            partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
            receive.buffer.bytes = 32768
            reconnect.backoff.max.ms = 1000
            reconnect.backoff.ms = 50
            request.timeout.ms = 30000
            retries = 2147483647
            retry.backoff.ms = 100
            sasl.client.callback.handler.class = null
            sasl.jaas.config = null
            sasl.kerberos.kinit.cmd = /usr/bin/kinit
            sasl.kerberos.min.time.before.relogin = 60000
            sasl.kerberos.service.name = null
            sasl.kerberos.ticket.renew.jitter = 0.05
            sasl.kerberos.ticket.renew.window.factor = 0.8
            sasl.login.callback.handler.class = null
            sasl.login.class = null
            sasl.login.refresh.buffer.seconds = 300
            sasl.login.refresh.min.period.seconds = 60
            sasl.login.refresh.window.factor = 0.8
            sasl.login.refresh.window.jitter = 0.05
            sasl.mechanism = GSSAPI
            security.protocol = PLAINTEXT
            security.providers = null
            send.buffer.bytes = 131072
            ssl.cipher.suites = null
            ssl.enabled.protocols = [TLSv1.2]
            ssl.endpoint.identification.algorithm = https
            ssl.engine.factory.class = null
            ssl.key.password = null
            ssl.keymanager.algorithm = SunX509
            ssl.keystore.location = null
            ssl.keystore.password = null
            ssl.keystore.type = JKS
            ssl.protocol = TLSv1.2
            ssl.provider = null
            ssl.secure.random.implementation = null
            ssl.trustmanager.algorithm = PKIX
            ssl.truststore.location = null
            ssl.truststore.password = null
            ssl.truststore.type = JKS
            transaction.timeout.ms = 60000
            transactional.id = null
            value.serializer = class org.apache.kafka.common.serialization.StringSerializer
        
        [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.6.0
        [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 62abe01bee039651
        [main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1600385581070
        [kafka-producer-network-thread | producer-1] ERROR org.apache.kafka.common.utils.KafkaThread - Uncaught exception in thread 'kafka-producer-network-thread | producer-1':
        java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/JsonNode
            at org.apache.kafka.common.requests.ApiVersionsRequest$Builder.<clinit>(ApiVersionsRequest.java:36)
            at org.apache.kafka.clients.NetworkClient.handleConnections(NetworkClient.java:910)
            at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:555)
            at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:325)
            at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:240)
            at java.lang.Thread.run(Thread.java:748)
        Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.JsonNode
            at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
            at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
            ... 6 more
        [main] INFO org.apache.kafka.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
        
        Process finished with exit code 0

So, this all is my code and i am trying to create a kafka producer but i am getting error as above. so please anyone can assist me to resolve this problem

Anemology

looks like you are having dependency conflicts. Could you remove your dependencies to spark-sql-kafka-0-10_2.12 and slf4j-simple and try to run your code again? – Fink

thanks for your support, but it doesn't work – Anemology

Answer 1

I was facing the same issue, by adding the below dependencies , the issue got resolved.

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.10.3</version>
</dependency>

Seaborne

Thanks, that solved my problem. Any explanation why that solves the issue? – Hyder

Adding this library solves the problem. As Kafka Producer is expecting that to be available in the classpath as a runtime dependency. – Military

Answer 2

In case of using Kubernetes pod, consider following suggested by Redpanda team:

replace the current snappy version with the native version. The recommended command:

apk add --no-cache gcompat 

Click for more details

Evasive