Limit Memory Usage of Kafka Streams

Question

Is there a way to limit or define the max memory usage of a kafka streams application? I have enabled caching with my state stores but when I deploy in Openshift I get OOM killed on my pods. I have checked I have no memory leakes and all my state store iterators are being closed.

I have updated my RocksDbConfigSetter to the recommendations found in https://github.com/facebook/rocksdb/wiki/Setup-Options-and-Basic-Tuning#other-general-options with no luck.

When I look at the state store directory the size is about 2GB. Currently have 50GB of memory allocated to the application and it still OOMs

Zerla

Atm, it's not possible to define a strict limit on memory usage. – Harlin

So am I suppose to give my openshift more and more memory until it does not OOM? I am already at 20GB. Does startup (rebuilding the statestore) use more memory. App was running fine and when I went to restart the pod it OOM. – Zerla

It looks like there might be more to your problem than what you summarized here. – Whacky

Try to estimate your required memory : docs.confluent.io/current/streams/sizing.html – Harlin

I am using the defaults so its 112mb * 20 partitions and I am using a key value store. So that is 2240 mb. I am confused, is that in addition to the memory usage of actual data as described by scenario 3. – Zerla

I am encountering the same scenario even with RocksDB tuning? Is there any soft limit of memory or setting the heap size "-Xms -Xmx" can help? In scenario 3 from documentation, if we have state store size 20GB, do we need 20GB memory to hold that data in memory? – Lunation

Hey @MatthiasJ.Sax, has the situation changed? Can we set a strict memory usage? – Tutor

I guess :) -- issues.apache.org/jira/browse/KAFKA-8215 -- issues.apache.org/jira/browse/KAFKA-8324 -- issues.apache.org/jira/browse/KAFKA-8323 -- issues.apache.org/jira/browse/KAFKA-8637 -- Worth to upgrade to at least 2.3 – Harlin

@MatthiasJ.Sax We are also seeing memory issues in our environment and trying to implement RocksDBConfiSetter. #65814705 – Zapata

@Zerla Were you able to resolve the issue? – Handoff

Answer 1

Calculate how much state-stores are getting created for your application and the total size of application

Use

StreamsConfig.topicPrefix(TopicConfig.RETENTION_BYTES_CONFIG)

It will limit the data stored by the statestore as well as the statestore of the streaming application is saved in kafka topics as well

Orcinol