Kafka Configuration for only seeing last 5 minutes of data

Question

Sorry i am new in Kafka and this question migth be so easy but i need some help. i did not figure out some configurations. There is a stream data, i want Consumers to see only last 5 minutes of messages that procuders sent. I am using Confluent.Kafka for .Net,

var config = new Dictionary<string, object>{
                {"group.id","Test1Costumers"},
                {"bootstrap.servers",brokerEndpoint},
                { "auto.commit.interval.ms", 60000},
                { "auto.offset.reset", "earliest" }
            };

Here is config dictionary of Consumers in github example, another issue is i dont want to store messages in a topic more than 5 minutes cos i wont need those records if they are older than 5 minutes.

When i configure server.properties;

# The minimum age of a log file to be eligible for deletion due to age
log.retention.ms=60000

after a minute its throw error that file is currently uses

Thank you for your help.

Perrone

are you under windows? can you post the log? – Postdiluvian

Yes i am under windows but i dont know where is log. I need to get the message produced in last 5 minutes. I will launch a job to consume the message produced 5min. e.g., if current time is 11:15, I will consume the message between 11:10 and 11:15. That means I need to get start offset by time 11:10 and end offset by time Current time. When i start a new consomer its give me all messages from beggining. – Perrone

Answer 1

In Kafka server.properties there's a setting called log.segment.bytes, which is set to 1GB by default. Once a log segment has reached 1GB, it is closed, and only after that the retention kicks in. E.g. if you are producing 100MB of message per day, and your retention is 1 week, you'd actually retain the data for around 17 days before it gets deleted. That's because the log segment will take 10 days to be full (1GB) and from that time retention will kick in. In your case, I'm assuming you haven't changed the value for log.segment.bytes, but your retention is very low. So, it won't be able to clean up the data as the log segment is not yet closed.

Pruett