According to Kafka's commitTransaction documentation, commitTransaction will fail with TimeoutException if it doesn't receive a response in certain time
Note that this method will raise TimeoutException if the transaction cannot be committed before expiration of max.block.ms. Additionally, it will raise InterruptException if interrupted. It is safe to retry in either case, but it is not possible to attempt a different operation (such as abortTransaction) since the commit may already be in the progress of completing. If not retrying, the only option is to close the producer.
Consider an application in which a Kafka producer sends a group of records as Transaction A.
After the records have been successfully sent to the topic, Kafka producer will execute commitTransaction .
Kafka cluster receives the commit transaction request and successfully commits records that are part of transaction A. Kafka cluster sends an acknowledgement regarding successful commit.
However, due to some issue this acknowledgement is lost, causing a Timeout exception at Kafka producer's commitTransaction call. Thus even though the records have been committed on Kafka cluster, from producer's perspective the commit failed.
Generally in such a scenario the application would retry sending the transaction A records in a new transaction B, but this would lead to duplication of records as they were already committed as part of transaction A.
Is the above described scenario possible?
How do you handle loss of commitTransaction acknowledgement and the eventual duplication of records that is caused by it?