How to set the max size of a Kafka message using librdkafka

Question

I'm trying to use Kafka to send a message of ~10Mb. I know its default size is 1Mb, but is that a hard limit? Can librdkafka support >10Mb and how do I set it?

Thunderstruck

Answer 1

You need to configure the topic max.message.bytes (see https://kafka.apache.org/22/documentation.html#topicconfigs), and configure the producer with message.max.bytes (see https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md). Please allow some extra space (at least ~500 bytes) for protocol overhead.

Candlestand

hey thanks for the reply. I tried doing this, and I no longer received the error stating that the message was too big. But it seems to hang now. So what I did was I modified the preexisting topic via the scripts in you reply, I also created a brand new one just to be safe. Both of which using 15Mb size (to be extra safe). Now when i modify the code to invoke the max change, it doesn't error anymore but everything above 1Mb will just stop. The producer is running on its own thread so i can still do things and try to send more messages. But none are going through – Thunderstruck

A bit of reconnaissance shows that I can successful send 9998999 bytes (9.9 Mb and its pretty fast too). But anything more than that follows the following comment where it just hangs and no longer produces @Candlestand – Thunderstruck

Reproduce this with debug=msg,protocol configured on the producer and then open an issue here: github.com/edenhill/librdkafka/issues/new Make sure you are using the latest version of librdkafka (v1.3.0 or v1.4.0-RC4) – Candlestand

Hey @Edenhill, wasn't familiar with your debug stuff so when i ran it i received a bunch of: No more space in current MessageSet, but when i googled it, all i got was references to your github... so maybe I am not using the most up to date, i thought. I used apt to install from a repo and although had your source never compiled it directly. Well doing that and linking, fixed it all. Evidently i wasn't using the most up-to-date. Also your name appears a lot so i think you are 'the guy'. Your code has been a pleasure to look through and the documentation is among the best. thank you! – Thunderstruck

@Candlestand "Please allow some extra space (at least ~500 bytes) for protocol overhead." -> THIS is useful. I was wondering why with all limits set to 1 MB I cannot send a message with the size of 999908 bytes, but 999907 was OK :) – Bahena

Fun fact: if I send a message to a topic with a shorter name, the size of the message can be greater than aforementioned 999907 bytes. Obviously, the topic name counts for this "protocol overhead". – Bahena