How to get rid of negative consumer lag in Kafka

Question

Recently I monitor my cluster and I saw the negative consumer lag on it:

This negative lag affects the sink connectors on the cluster and I got this error on the Kafka-Connect:

Mar 25 15:56:30 Master1 connect-distributed[14530]: [2020-03-25 15:56:30,910] WARN WorkerSinkTask{id=hdfs-1} Ignoring invalid task provided offset mongodb.db.collection-1/OffsetAndMetadata{offset=84, leaderEpoch=null, metadata=''} -- not yet consumed, taskOffset=84 currentOffset=7 (org.apache.kafka.connect.runtime.WorkerSinkTask:417)

Zehe

"Producer offset is polled. Consumer offset is read from the offset topic for Kafka based consumers. This means the reported lag may be negative since we are consuming offset from the offset topic faster then polling the producer offset. This is normal and not a problem." Taken from github.com/yahoo/CMAK. I think same case happens in here too. – Frontwards

Thanks. so if it is normal. what's the reason that kafka-connect return this error? – Zehe

I don't think these are somehow related. But don't really know what is the reason of your problem. – Frontwards

Not clear how your data is being gathered... Have you tried Burrow or Remora instead? – Hassle