Kafka connect HDFS sink ERROR failed creating a WAL

Question

I'm using Kafka connect HDFS. When I'm trying to run my connector I'm got the following exception:

ERROR Failed creating a WAL Writer: Failed to create file[/path/log] for [DFSClient_NONMAPREDUCE_208312334_41] for client [IP] because this file is already beign create by [DFSClient_NONMAPREDUCE_165323242_41]

Any suggestions please?

Cheston

Did you solve it? – Overspread

Answer 1

As per further testing, upgrading the version to 2.7.7 didn't help. We still ran into WAL issue. We ended using a different sink s3SinkConnector.

Edme

I think you need to describe more details of what you did and at least one reason for doing only that. Like the command to clean up. – Surrender

As per further testing, upgrading the version to 2.7.7 didn't help. We still ran into WAL issue. We ended using a different sink s3SinkConnector. – Edme

Answer 2

Aside from the workaround to cleanup the wal logs and connector restart. We are also testing on upgrading the hadoop local version 2.73 to 2.7.7 and the kafka-hdfs version to 5.5 of our sink connector from Its currently on testing and so far the wal issue is not recoccuring.

Edme