Kafka - Broker fails because all log dirs have failed

Question

I am attempting a simple kafka config on Windows os. My zookeeper and kafka installations use default configs (except for data and log dir paths). I can start kafka and produce/consume messages without issue; however, when the broker attempts to delete old messages (I set log retention to 100 ms), I get the following error:

    java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
        at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
        at java.nio.file.Files.move(Files.java:1395)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:697)
        at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:212)
        at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:415)
        at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:1601)
        at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:1588)
        at kafka.log.Log$$anonfun$deleteSegments$1$$anonfun$apply$mcI$sp$1.apply(Log.scala:1170)
        at kafka.log.Log$$anonfun$deleteSegments$1$$anonfun$apply$mcI$sp$1.apply(Log.scala:1170)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at kafka.log.Log$$anonfun$deleteSegments$1.apply$mcI$sp(Log.scala:1170)
        at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161)
        at kafka.log.Log$$anonfun$deleteSegments$1.apply(Log.scala:1161)
        at kafka.log.Log.maybeHandleIOException(Log.scala:1678)
        at kafka.log.Log.deleteSegments(Log.scala:1161)
        at kafka.log.Log.deleteOldSegments(Log.scala:1156)
        at kafka.log.Log.deleteRetentionMsBreachedSegments(Log.scala:1228)
        at kafka.log.Log.deleteOldSegments(Log.scala:1222)
        at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:854)
        at kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:852)
        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
        at kafka.log.LogManager.cleanupLogs(LogManager.scala:852)
        at kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:385)
        at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
        at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:62)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.

                at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
                at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
                at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
                at sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
                at java.nio.file.Files.move(Files.java:1395)
                at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)
                ... 32 more

And then...

[2018-08-01 18:14:01,479] INFO [ReplicaManager broker=0] Stopping serving replicas in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs (kafka.server.ReplicaManager)
[2018-08-01 18:14:01,480] ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.KafkaStorageException: Error while deleting segments for discrete-topic-0 in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs
Caused by: java.nio.file.FileSystemException: C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log -> C:\Workspace\kafka_2.11-1.1.0\kafka-logs\discrete-topic-0\00000000000000000000.log.deleted: The process cannot access the file because it is being used by another process.
[2018-08-01 18:14:01,504] INFO Stopping serving logs in dir C:\Workspace\kafka_2.11-1.1.0\kafka-logs (kafka.log.LogManager)
[2018-08-01 18:14:01,508] ERROR Shutdown broker because all log dirs in C:\Workspace\kafka_2.11-1.1.0\kafka-logs have failed (kafka.log.LogManager)

This seems to be this issue: https://issues.apache.org/jira/browse/KAFKA-6188

kafka version: kafka_2.11-1.1.0 Windows 10 Zookeeper version: 3.4.12

I have a few questions:

1. Has anyone seen this issue and is there a workaround?
2. Is running a kafka broker on Windows a viable options at this point? I know it is not supported for Windows and there seem to be multiple critical issues.

Thanks

Gymnosperm

Any reason you're not using the latest Kafka? And 100ms is really small... But anyway, that file is still being used by another process, and that's not exactly a windows issue – Alkyd

You can check out my answer on this post: https://mcmap.net/q/375324/-kafka-configuration-for-only-seeing-last-5-minutes-of-data/… I think it's the same issue. – Scarab

1) yes and no workaround 2) There are no tests for windows so it is not really a supported platform so no go for production from my point of view – Judaea

1) I set retention to 100 ms to quickly clear the log. I agree this is a very small vallue and not for normal operation. 2) @ Bitswazsky Your solution does seem to help. Thank you – Gymnosperm

Answer 1

step 1 : point to a new log directory in server.properties file and save the file

log.dirs=C:\Tools\kafka_2.11-2.1.0\kafka-test-logs

step 2 : start the kafka server again

/bin/windows/kafka-server-start.bat /config/server.properties

Kirtle

That (changing log.dirs) helped, although unclear why – Pearlinepearlman

It is strange. After log.dirs=C:\Tools\kafka_2.11-2.1.0\kafka-test-logs created directory c:\ProgramPortable\kafka_2.12-2.3.0\Toolskafka_2.11-2.1.0kafka-test-logs\ Is it what you expected? – Pentadactyl

You have to use \\ to escape the backward slashes. – Hassan

This resolved the error but does it remove messages from existing topics? – Temporize

Answer 2

I had the same kafka behavior but simply renaming or cleaning log directories wouldn't do the trick. And every time I tried to start kakfa it would crash (it was generating back all the logs deleted).

Solution:

1. Delete the logs as before.
2. Go to ZooKeeper data dir (you find it in \conf\zoo.cfg) and clean it. Restart everything and it should be completely reset.

Diarrhoea

I have also deleted zookeeper data logs and it worked – Oneupmanship

Answer 3

You can delete all Kafka data. See Kafka docs:

1. Stop the producer and consumer clients with Ctrl-C, if you haven't done so already.
2. Stop the Kafka broker with Ctrl-C.
3. Lastly, stop the ZooKeeper server with Ctrl-C.

If you also want to delete any data from your local Kafka environment including any events you have created along the way, run the command:

$ rm -rf /tmp/kafka-logs /tmp/zookeeper

If you are on a Windows machine, you can use the following command

rmdir /s /q C:\tmp\kafka-logs

rmdir /s /q C:\tmp\zookeeper

Gramarye

Answer 4

Go to config folder in kafka folder (e.g. D:\kafka\config)  open zookeeper.properties Edit your datadir to look like : dataDir = D:/kafka/data/zookeeper open server.properties in notepad and edit logs.dirs and zookeeper.connect logs.dirs = D:/kafka/data/kafka zookeeper.connect=localhost:2181 NOTE: replace D:\kafka according to your own settings and folders you have for kafka

Kenakenaf

Answer 5

I am using Kafka 2.12-2.3.0 on Windows. And What worked for me was changing server.properties setting log.dirs= to contains both log data Kafka and Zookeeper.

Example :

log.dirs=C:/kafka_2.12-2.3.0/data/kafka, C:/kafka_2.12-2.3.0/data/zookeeper

Salientian

Is this applicable when using zookeeper which came with kafka – Potentiate

Thanks! This worked for me. I'm using Windows 10 with Kafka 2.13. – Jeopardy

Answer 6

I faced similar issue when I run zookeeper, kafka server, command line producer and consumer locally with default configs as described here.

Server and producer started successfully, but when I tried to start consumer I immediately got the issue with failed logs.

I fixed it by setting different log directories for server, producer and consumer. As far as there is no explicit way to set it for producer and consumer I had to update their start scripts. I added a line

set LOG_DIR=/distr/kafka_2.11-2.2.0/producer_logs

to kafka_console_producer.bat
And /distr/kafka_2.11-2.2.0/consumer_logs for kafka_console_consumer.bat respectively.
After the fix consumer successfully started. Seems the problem is because all three processes use the same log directory and interfere with each other somehow.

Wrung

Answer 7

Delete the log files of Kafka brokers and also the data and log files of zookeeper and restart both. Also keep in mind to stop the Kafka clients before you restart the Kafka and Zookeeper

Augend

Answer 8

I tried multiple methods to solve this issue:

1. deleting logs from folder pointed by log.dirs property in server.properties. it does solve my problem for certain duration but it stoped working when i had more than 20 topics created so each time i start my kafka server this issue was coming.

2. tried deleting topics from kafka using command "kafka-topics --zookeeper localhost:2181 --delete --topic topicname" but it marked my topic as --"marked to be deleted flag". but did not solved my problem.

then finally i tried deleting folders zookeeper and kafka-logs inside tmp folder of kafka and it worked and all topics were deleted.

Bendigo

Answer 9

Try creating a new folder inside the Kafka folder. Change server.properties and zookeeper.properties and restart the command prompts and again try starting the servers.

Hachure

Answer 10

In dev machine, I stopped zookeeper and kafka-server and all client and deleted the contents of the logs directory. Then started the zookeeper, kafka, connectors, clients in sequence.

Puttier

Answer 11

The problem is because of that you are running out of storage for logs.

Change log.dirs in server.properties. Also make sure that you and then restart Kafka.

P.S. 1: By doing so, the older messages will be lost.

P.S. 2: Don't set the tmp/ as log.dirs location, since it is purged once your machine turns off.

Dupree

Answer 12

As all the answers have parts of correct points, but none of them was a complete solution for me, I will add what helped me:

1. Stop both - Kafka and ZooKeeper;
2. Locate Kafka logs directory (value of the log.dirs variable in KAFKA_HOME\config\server.properties file) and clean it;
3. Locate ZooKeeper data directory (value of the data.Dir variable in KAFKA_HOME\config\zookeeper.properties file) and clean it;
4. Restart ZooKeeper;
5. Restart Kafka.

Enucleate

Answer 13

Remove all files under the local folder as shown in your docker monitor. After, restart the containers.

Galvez

Answer 14

Similar to what Tutor did above, what worked for me was deleting the zookeeper-data folder. I'm on Windows 10, so the zookeeper-data folder was at my root Kafka folder, and defined in config/zookeeper.properties

dataDir=c:/projects/kafka/zookeeper-data

FYI..you'll lose whatever topics, etc. you've already created

Der

Answer 15

chown kafka -R /home/kafka/kafkalogs

worked for me

Barker