Kafka Connect running out of heap space

Question

After starting Kafka Connect (connect-standalone), my task fails immediately after starting with:

java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:343)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:291)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:232)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:180)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:193)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:248)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1013)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:979)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.pollConsumer(WorkerSinkTask.java:316)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:222)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:170)
    at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:142)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

There's a mention of heap space in some Kafka documentation, telling you to try it with "the default" and only modifying it if there are problems, but there are no instructions to modify the heap space.

Hypso

The "default size" seems to be determined at run time. It appears to be large enough in my machine (8G). I still get the OOM error. Besides, there are all sorts of other exceptions in the connector log, and the server stalls. Very frustrating. – Horace

From what I found, it's hard-coded to 256M: github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L209 – Hypso

ohhh! thank you. That explains. – Horace

Answer 1

You can control the max and initial heap size by setting the KAFKA_HEAP_OPTS environment variable.

The following example sets a starting size of 512 MB and a maximum size of 1 GB:

KAFKA_HEAP_OPTS="-Xms512m -Xmx1g" connect-standalone connect-worker.properties connect-s3-sink.properties

When running a Kafka command such as connect-standalone, the kafka-run-class script is invoked, which sets a default heap size of 256 MB in the KAFKA_HEAP_OPTS environment variable if it is not already set.

Hypso

As of Kafka 1.1.0, the new default for Connect is -Xmx2g, and it is set before kafka-run-class - github.com/apache/kafka/commit/… – Jejune

Any other way to achieve the same? – Maplemaples

The method in my answer is the same, only the default value changed. – Hypso

Answer 2

When you have Kafka problems with

java.lang.OutOfMemoryError: Java heap space

it doesn't necessarily mean that it's a memory problem. Several Kafka admin tools like kafka-topics.sh will mask the true error with this when trying to connect to an SSL PORT. The true (masked) error is SSL handshake failed !

See this issue: https://issues.apache.org/jira/browse/KAFKA-4090

The solution is to include a properties file in your command (for kafka-topics.sh this would be --command-config) and to absolutely include this line:

security.protocol=SSL

Hugmetight

for my case, it does appear to be a bogus error message, hiding the actual issue – Akers

Wow. This helped! I only had producer: ssl: key-store-location: and password: configured in application.yaml. And I was getting out of memory in testing environment. I think somewhere in the error log there was also Topic not present in metadata after 60000 ms. Adding spring.kafka.producer.security.protocol: SSL helped. Thank you. – Pomp

You're spot on! Thank you very much indeed. Two years on and this is still happening with Spring Boot 2.7.2 and Kafka 2.8. – Bankbook

Hi, a bit late, but I am facing the same issue even with SSL enabled! any hint? im setting kafkaheapsize to -Xms3G -Xmx3G which is, I suppose, enough for my case. – Holbert

I am also getting the below error when I am attempting to start kafka on my mac ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.OutOfMemoryError: Java heap space – Chronologist

same here when playing with kafka tools connecting to an AWS MSK cluster... for me was that kafka-topics.sh was not called with "--command-config client.properties" where security.protocol=SASL_SSL – Lunulate

Answer 3

I found another cause of this issue this morning. I was seeing this same exception except that I'm not using SSL and my messages are very small. The issue in my case turned out to be a misconfigured bootstrap-servers url. If you configure that URL to be a server and port that is open but incorrect, you can cause this same exception. The Kafka folks are aware of the general issue and are tracking it here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-498%3A+Add+client-side+configuration+for+maximum+response+size+to+protect+against+OOM

Convolve

Yes, after adding sasl client security protocols using --command-config it got worked. – Fulminant

Answer 4

Even I was facing the issue could not start my producer and consumer for a given topic. Also deleted all unnecessary log files and topics.Even though that's not related to the issue.

Changing the kafka-run-class.sh did not work for me. I changed the below files

kafka-console-consumer.sh

kafka-console-producer.sh

and stopped getting OOM error. Both consumer and producer worked fine after this.

Increased the size to KAFKA_HEAP_OPTS="-Xmx1G" was 512m earlier.

Pretty

Sorry that my answer wasn't more clear. KAFKA_HEAP_OPTS is an environment variable and should be set at the command line or in the service that starts the Kafka job. You should not modify the scripts that are part of the Kafka distribution, as they'll be wiped out when you update Kafka. – Hypso

Answer 5

In my case, using a Spring Boot 2.7.8 application leveraging Spring Boot Kafka auto-configuration (no configuration in Java code), the problem was caused by the security protocol not being set (apparently the default value is PLAINTEXT). Other errors I got together with java.lang.OutOfMemoryError: Java heap space are:

Stopping container due to an Error
Error while stopping the container: 
Uncaught exception in thread 'kafka-producer-network-thread | producer-':

The solution was to add the following lines to my application.properties:

spring.kafka.consumer.security.protocol=SSL
spring.kafka.producer.security.protocol=SSL

My attempt to fix it with just:

spring.kafka.security.protocol=SSL 

did not work.

Michey

Answer 6

for windows

bin\windows\kafka-server-start.bat

for linux/mac

bin\kafka-server-start.sh

update these values

set KAFKA_HEAP_OPTS=-Xmx2G -Xms2G

Georgettageorgette