Apache Kafka - Resetting the last seen epoch of partition. Why?

Question

We use Kafka Mirror Maker Version 1 to mirror data between Kafka Clusters. I know that MM1 is deprecated but it is a solid pice of software that does exactly what we need. We use it out of a dedicated Kafka installation that is independent of the clusters between which we store data (which are running Kafka Version 2.6 by now).

We used Kafka 2.7.x for the "MirrorMaker-Kafka" and recently updated this one to 3.3.1. Since then we have a lot of the following messages in out MM log:

2022-12-01 15:07:45,368 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:12:45,373 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:22:45,388 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:37:45,394 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:42:45,398 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw

As you can see the message is repeated every 5 Minutens. We mirror several hundret partitions and the message is logged for many (maybe all) partitions.

As I could found out the message was introduced in the KAFKA-12257 fix in this commit.

Unfortunately, the meaning of the message is unclear to me. The constant repetition at a certain interval also makes me wonder. Possibly I still have weaknesses in my (Producer) configuration.

If someone could explain the phenomenon and know what measures I can take to improve it, I would be very pleased.

Martelli

What was the conclusion about the error? I am also facing the similar issue and couldn't able to understand the reason. – Sawhorse

Same for me... let me know if you found a solution – Decease

Answer 1

The message that you are seeing is related to the KAFKA-12257 fix, which was implemented to address an issue with the producer client not properly tracking the epoch of a topic partition. Specifically, the fix ensures that the producer client is able to keep track of the epoch of a partition by associating a unique identifier (the "topicId") with it.

In your case, the message is being logged every 5 minutes because the producer client is resetting the last seen epoch of a specific partition (MyTopic-5) to 87, since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw.

To ensure that the producer is able to properly track the epoch of a partition, you should review your producer configuration and ensure that the "topicId" is set properly. Additionally, you should also ensure that your producer client is configured to properly handle partition reassignment events, as this can cause the topicId to change.

Finally, if you are still having issues with the producer client not properly tracking the epoch of a partition, you may want to consider upgrading to a newer version of Kafka Mirror Maker, as MM1 is now deprecated.

Torrietorrin

Thanks for clarification. I just wonder where I (as a producer) can influence the topicId. In the Kafka documentation and the Producer API there is hardly anything about the "topicId". To me it looks like an internal concept that I - as an API user - don't come into contact with. Also resetting the last seen epoch every 5 minutes from "null" to the same topicId looks strange as nothing changes with regard to the topicId. It doesn't look like a MM1 problem to me either. There is only the usual producer API used. – Martelli

Maybe the version gap between server (2.6.x) and Client (3.3.1) is to big. Although it is expected to work because of Bidirectional Client Compatibility. – Martelli

Would you elaborate more to comment on "producer configuration and ensure that the "topicId" is set properly", I have reviewed documentation, but haven't found anything about: docs.confluent.io/platform/current/installation/configuration/… – Have

Answer 2

according to https://forum.confluent.io/t/resetting-the-last-seen-epoch-of-partition/6740/2:

When a client first try to consume/produce from/to a topic, it will initialise an object in memory to track the topic metadata, and upon the first metadata response received from the kafka cluster.It will update its state from the default null to the actual topic ID.

that makes it sound like this error can be safely ignored, as long as it is changing from null, rather than some other value, in which case i would look at it in more detail.

Flowerdeluce

Answer 3

May be consumer group Id that you are using which is not having access to your topic .Please check in kafka logs .

Telmatelo