.NET Confluent Kafka consumer memory leak

Question

We're seeing huge memory leaks when consuming from Kafka using the Confluent Kafka .NET library.

One thing I've noticed is the code consumes without a using statement:

while (true)
{
    if (_consumer.Consume(out Message<string, string> message, TimeSpan.FromMilliseconds(100)))
    {
        OnMessage(message);
    }
}

However, that while loop runs for the lifetime for the application, so .Dispose() would never be called whilst consuming anyway. No other consumer instances are created.

As the code behind the library is in C, will objects created by the library be cleaned if we call GC.Collect(), or is this unmanaged code which the garbage collector has no control over?

Anything else which might be causing the leak, does consumer.Close() need to be called at certain periods or something like that?

Daguerre

You're right, no need to call to .Dispose, generally you would want to call .Dispose or .Close to notify the server that the consumer is no longer available, but it has nothing to do with memory management, at least not during the runtime of the application. How long did you let the application run? maybe the GC just hasn't kicked in yet.. Try to use a workstation GC and maybe run a profiler. – Lewse

Did you ever learn anything about this? I might be seeing the same thing. – Smithereens