java.nio.file.Files.delete(Path path) - occasional failure to recursively delete directory using SimpleFileVisitor

Question

Trying to troubleshoot an occasional java.nio.file.DirectoryNotEmptyException in a recursive delete method taken from Delete directories recursively in Java

Code (credit to @TrevorRobinson) :

static void removeRecursive(Path path) throws IOException {
    Files.walkFileTree(path, new SimpleFileVisitor<Path>() {

        final Logger logger = LoggerFactory.getLogger(this.getClass());
        @Override
        public FileVisitResult visitFile(Path file,
                BasicFileAttributes attrs) throws IOException {
            logger.warn("Deleting " + file.getFileName());
            Files.delete(file);
            logger.warn("DELETED " + file.getFileName());
            return FileVisitResult.CONTINUE;
        }

        @Override
        public FileVisitResult visitFileFailed(Path file, IOException exc) {
            // try to delete the file anyway, even if its attributes could
            // not be read, since delete-only access is theoretically possible
            // I NEVER SEE THIS
            logger.warn("Delete file " + file + " failed", exc);
            try {
                Files.delete(file);
            } catch (IOException e) {
                logger.warn(
                    "Delete file " + file + " failed again", exc);
            }
            return FileVisitResult.CONTINUE;
        }

        @Override
        public FileVisitResult postVisitDirectory(Path dir, IOException exc)
                throws IOException {
            if (exc == null) {
                Files.delete(dir);
                return FileVisitResult.CONTINUE;
            }
            // directory iteration failed; propagate exception
            throw exc;
        }
    });
}

Call:

try {
    removeRecursive(Paths.get(unzipDirPath));
} catch (IOException e) {
    String msg = "Failed to delete folder " + unzipDirPath;
    if (e instanceof java.nio.file.DirectoryNotEmptyException) {
        msg += ". Still contains : ";
        final File[] listFiles = Paths.get(unzipDirPath).toFile().listFiles();
        if (listFiles != null) for (File file : listFiles) {
            msg += file.getAbsolutePath() + "\n";
        }
    }
    log.error(msg, e);
}

Prints (once in 20/40 iterations):

22:03:34.190 [http-bio-8080-exec-47] WARN  g.u.d.m.server.servlets.Controller$1 - Deleting batt
22:03:34.192 [http-bio-8080-exec-47] WARN  g.u.d.m.server.servlets.Controller$1 - DELETED batt
22:03:34.192 [http-bio-8080-exec-47] WARN  g.u.d.m.server.servlets.Controller$1 - Deleting wifi
22:03:34.193 [http-bio-8080-exec-47] WARN  g.u.d.m.server.servlets.Controller$1 - DELETED wifi
22:03:34.196 [http-bio-8080-exec-47] ERROR g.u.d.m.s.s.DataCollectionServlet - Failed to delete folder C:\yada\. Still contains : C:\yada\dir\wifi

java.nio.file.DirectoryNotEmptyException: C:\yada\dir
    at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:265) ~[na:1.7.0_45]
    at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) ~[na:1.7.0_45]
    at java.nio.file.Files.delete(Files.java:1077) ~[na:1.7.0_45]
    at gr.uoa.di.monitoring.server.servlets.Controller$1.postVisitDirectory(Controller.java:128) ~[Controller$1.class:na]
    at gr.uoa.di.monitoring.server.servlets.Controller$1.postVisitDirectory(Controller.java:1) ~[Controller$1.class:na]
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:224) ~[na:1.7.0_45]
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:199) ~[na:1.7.0_45]
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69) ~[na:1.7.0_45]
    at java.nio.file.Files.walkFileTree(Files.java:2600) ~[na:1.7.0_45]
    at java.nio.file.Files.walkFileTree(Files.java:2633) ~[na:1.7.0_45]
    at gr.uoa.di.monitoring.server.servlets.Controller.removeRecursive(Controller.java:96) ~[Controller.class:na]
    at gr.uoa.di.monitoring.server.servlets.DataCollectionServlet.doPost(DataCollectionServlet.java:153) ~[DataCollectionServlet.class:na]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:641) [servlet-api.jar:na]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) [servlet-api.jar:na]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) [catalina.jar:7.0.32]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) [catalina.jar:7.0.32]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) [catalina.jar:7.0.32]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) [catalina.jar:7.0.32]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) [catalina.jar:7.0.32]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) [catalina.jar:7.0.32]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) [catalina.jar:7.0.32]
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929) [catalina.jar:7.0.32]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) [catalina.jar:7.0.32]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) [catalina.jar:7.0.32]
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1002) [tomcat-coyote.jar:7.0.32]
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585) [tomcat-coyote.jar:7.0.32]
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) [tomcat-coyote.jar:7.0.32]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

Notice that wifi is reported as deleted - what's even more weird is that sometimes I get :

Failed to delete folder C:\yada. Still contains : C:\yada\dir

java.nio.file.DirectoryNotEmptyException: C:\yada\dir

I tend towards the conclusion that occasionally the deletion takes too long - in other words the problem is that java.nio.file.Files.delete(Path path) does not block (so C:\yada\dir still contains files when its time comes, which sometimes are deleted by the time I stat it). So how am I to workaround this ?

Also : is java.nio.file.Files.delete(Path path) required to throw ? The docs state :

On some operating systems it may not be possible to remove a file when it is open and in use by this Java virtual machine or other programs.

Does not seem to require an exception to be thrown in this case. Is java.nio.file.Files.delete(Path path) required to throw ?

Linlithgow

You might find some answers in this thread: #1729549 – Schistosome

@crownjewel82: That's for java.io.File.delete() which returns boolean ! – Linlithgow

It also throws different exceptions based on the problem: FileNotFoundException, SecurityException, etc. You can catch those exceptions and respond to each appropriately. – Schistosome

@crownjewel82: I am asking about a different method....... – Linlithgow

Both methods do the same thing: docs.oracle.com/javase/7/docs/api/java/nio/file/… – Schistosome

Bizarre fun fact: I experienced similar behavior - for a directory with the same name as in OP's question - wifi. I could consistently reproduce the issue for a specific directory tree and it would always fail for a directory named wifi on Windows. In my case, the solution was to rely on WatcherService for deletion confirmation events from the file system itself. However, I was already leveraging WatcherService in another part of the app - the timing issue may have actually been caused by this API due to dir locks. – Congener

Answer 1

I was having the same problem and it turned out the problem was caused by an unclosed directory file stream somewhere else in the code for the same directory I was deleting stuff from. The stream object returned by:

Files.list(Path)

must be closed, so be sure to use the try-with-resources construct in your code if using that method.

So I don't think it's that the deletion takes too long, I tried a wait before re-attempting directory deletion without any luck. It's most likely that your own program has that resource locked. The result is that the delete call on it doesn't complete although it returns successfully (it looks like Windows will eventually delete the file once your own program frees it) but then of course the containing directory cannot be deleted as that is indeed not yet empty.

Cyder

I am usually careful with this kind of stuff - with " I tried a wait before re-attempting directory deletion without any luck " you mean that the method did not fail despite the wait ? In this case it might be that (file was not closed) although I can't be sure as the code was a long time ago - will look at it again though – Linlithgow

It is plain silly, that terminal operations on a Stream<T> does not call close() on it. Using try-with-resources on my previous Files.list(path) solved this same issue for me. – Breechcloth

Thanks, that is easy to overlook. After 30 mins of debugging, it turned out this was the cause. – Keratogenous

Answer 2

I know this is a very old thread - but I had the same problem and it took me quite some time to get it fixed. I think this misbehaviour is caused by a timing issue (looks like it happens on Windows only), so I put a pause into the postVisitDirectory method. That worked and this is what I finally came up with:

A method that does the delete without throwing the DirectoryNotEmptyException:

private boolean isDeleted(Path dir) throws IOException {
    boolean deleted = false;
    try {
        Files.delete(dir);
        deleted = true;
    } catch (DirectoryNotEmptyException e) {
    // happens sometimes if Windows is too slow to remove children of a directory
    deleted = false;
    }
    return deleted;
}

and its use in a loop:

public FileVisitResult postVisitDirectory(Path dir, IOException e) throws IOException {
    if (e == null) {
        int maxTries = 5;
        int count = 0;
        boolean deleted = false;
        do {
            if ((deleted = this.isDeleted(dir))) {
                break;
            } else {
                // wait a bit and try again
                count++;
                try {
                    Thread.sleep(1);
                } catch (InterruptedException e1) {
                    Thread.currentThread().interrupt();
                    break;
                }
            }
        } while (count < maxTries);
        // gone?
        if (!deleted) {
            throw new DirectoryNotEmptyException(dir.toString());
        }
        // go ahead
        return FileVisitResult.CONTINUE;
    }
    throw e;
}

Andy

Watchword

Answer 3

I would have commented on the answer by @user3485962, but don't have enough points for that!

Here is an example using try with resources that closes the stream after creating a list:

    /**  
     * @param dir The directory to list.
     * @return A list of files and directories in dir.
     * @throws IOException If encountered.
     */
    public static List<Path> getList(Path dir) throws IOException {
        try (Stream<Path> s = Files.list(dir)) {
            return s.collect(Collectors.toList());
        }
    }

Panjabi

Here is a link to the answer that this answer relates to: https://mcmap.net/q/391771/-java-nio-file-files-delete-path-path-occasional-failure-to-recursively-delete-directory-using-simplefilevisitor – Panjabi

Answer 4

You could temporarily store the name of the file you're currently iterating in a variable and perform a try-catch on DirectoryNotEmptyException. When this exception occurs, catch it and throw your own exception that specifies the file.

try {
 Files.delete(file);
} catch (DirectoryNotEmptyException  e) {
 throw new MySpecificException(file.getFileName());
}

class MySpecificException extends Exception {
 public MySpecificException() { }

 public MySpecificException(string filename) {
  super(filename);
 }  
}

Getting the filename can be done with e.getMessage();

I assume Files.delete() continues deleting the files in a directory when it encounters a file that can't be deleted. If this is the case, my method still works: just return a list of all the files in the directory instead of the filename of the directory. It should only contain files that can't be deleted and you still have your solution.

Rickyrico

Off topic - the file you are catching is the directory that is not empty - the recursive method should delete all the files in the directory and this is failing - and I want to see which file failed to be deleted and why ! – Linlithgow

I want to know why as clearly stated in the question - I already have a list - see my edit. Gosh I am not even sure delete does not throw - the docs are unclear – Linlithgow

You might want to be more polite to people who try to help you. The DirectoryNotEmptyException is thrown because you're attempting to delete a folder that has files in it. This is programmed in a way that it simply skips the files that it can't delete: no events are raised and no exception is thrown. If you want to check the cause, you can use File.canRead(), file.CanWrite() and File.canExecute() to see if it's a permission problem and/or if it's in use. These should be the only reasons why a file can't be deleted. – Rickyrico

Please notice that you were not replying to the question - I find it not polite either to not read a question carefully (you might want to follow the revisions to see for instance that I say that I am able to list the files since edit 0 : stackoverflow.com/revisions/19935624/1- and you come along and reply to me that I can list the files, completely missing the point !). Where exactly was I rude ? Gosh went to the (unclear) docs. Anyway - back to the question : "This is programmed in a way that it simply skips the files that it can't delete" - which is this ? – Linlithgow

Answer 5

An expansion on my comment above.

When Java has a problem doing something it throws an Exception. Exceptions, like all other types, can be inherited from. The API will specify which checked exceptions each method throws. The methods in java.io and java.nio will usually throw IOException or one of it's children. If you want to create a method that will tell you why the file operation, in this case deletion, failed you can do something like this:

@Override
public FileVisitResult visitFileFailed(Path file, IOException exc) throws IOException {
       //You don't need to try to delete the file again since this method was called
       //because the deletion failed. Instead...
       if (exc instanceof NoSuchFileException) {
            System.out.println("Could not find the file: " + file);
       } else if (exc instanceof DirectoryNotEmptyException) {
            System.out.println("The directory [+ " file + "] was not empty.");
       } else {
            System.out.println("Could not delete file [" + file
                + "]. There was a problem communicating with the file system");
       }

       return FileVisitResult.CONTINUE;
}

You can change the actual program response as you need to but that should give you the general idea.

Schistosome

"You don't need to try to delete the file again since this method was called" --> wrong check the links in my answer. Also I have already done this - has nothing top do with my actual problem - which is : does delete throw if it fails ? The docs are not clear – Linlithgow

Out of curiosity, do you understand the difference between returns and throws? – Schistosome