Odd behaviour when deleting Files with Files.delete()

Question

Please consider the following example Java class (pom.xml below):

package test.filedelete;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.NoSuchFileException;
import java.nio.file.Path;

import org.apache.commons.io.IOUtils;

public class Main
{
    public static void main(String[] args) throws IOException
    {
        byte[] bytes = "testtesttesttesttesttesttesttesttesttest".getBytes();
        InputStream is = new ByteArrayInputStream(bytes);

        Path tempFileToBeDeleted = Files.createTempFile("test", "");
        OutputStream os = Files.newOutputStream(tempFileToBeDeleted);
        IOUtils.copy(is, os);

        deleteAndCheck(tempFileToBeDeleted);

        // breakpoint 1
        System.out.println("\nClosing stream\n");

        os.close();

        deleteAndCheck(tempFileToBeDeleted);
    }

    private static void deleteAndCheck(Path file) throws IOException
    {
        System.out.println("Deleting file: " + file);
        try
        {
            Files.delete(file);
        }
        catch (NoSuchFileException e)
        {
            System.out.println("No such file");
        }
        System.out.println("File really deleted: " + !Files.exists(file));

        System.out.println("Recreating deleted file ...");
        try
        {
            Files.createFile(file);
            System.out.println("Recreation successful");
        }
        catch (IOException e)
        {
            System.out.println("Recreation not possible, exception: " + e.getClass().getName());
        }
    }
}

I write to a FileOutputStream and try to delete the file afterwards without closing the Stream first. This was my original problem, and of course wrong, but it leads to some strange observations.

When you run the main Method on Windows 7 it produces the following output:

Deleting file: C:\Users\MSCHAE~1\AppData\Local\Temp\test6100073603559201768
File really deleted: true
Recreating deleted file ...
Recreation not possible, exception: java.nio.file.AccessDeniedException

Closing stream

Deleting file: C:\Users\MSCHAE~1\AppData\Local\Temp\test6100073603559201768
No such file
File really deleted: true
Recreating deleted file ...
Recreation successful

• Why does the first call to Files.delete() not throw an exception?
• Why does the following call to Files.exist() return false?
• Why is it not possible to create the file anew?

Regarding the last question I noticed that the file is still visible in the Explorer when you stop at breakpoint 1. When you terminate the JVM then, the file will be deleted anyway. After closing the stream deleteAndCheck() works as expected.

It seems to me that the deletion is not propagated to the OS before closing the stream and the Files API does not reflect this properly.

Can someone explain exactly what's happening here?

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>test</groupId>
    <artifactId>filedelete</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.4</version>
        </dependency>
    </dependencies>
</project>

Update for clarification

The file disappears in the Windows explorer, if the stream is closed AND Files.delete() is called - the last operation triggers -, or if Files.delete() has been called without closing the stream and the JVM is terminated.

Horripilation

What OS is it? If Unix-like, can you try and print the permissions of the parent directory? – Pomerania

And by the way, you can directly use Files.copy() to copy the contents of an InputStream into a Path; no need for IOUtils – Pomerania

Try putting a breakpoint on Files.delete(file); and then stepping past it. Does the file actually disappear from the explorer window at that moment? I'm suspicious that somehow the delete succeeds, but windows is keeping the file because there is another reference open to it. This may be something low level in the way windows works. I would not expect the same on a Posix (eg: Linux) system. – Blackmun

See the last two comments on dhke's answer. His, then mine. – Blackmun

Answer 1

Can you delete an open file?

It is perfectly valid to delete the directory entry of a file when the file is opened. In Unix this is the default semantics and Windows behaves similarily as long as FILE_SHARE_DELETE is set on all file handles open to that file.

[Edit: Thanks to @couling for discussions and corrections]

However, there is a slight difference: Unix deletes the file name immediately, while Windows deletes the file name only when the last handle is closed. It however, prevents you from opening a file with the same name until the last handle to the (deleted) file is closed.

Go figure ...

On both systems, however, deleting the file does not necessarily make the file go away, it still occupies space on the disk as long as there is still an open handle to it. Space occupied by the file is only released when the last open handle is closed.

Excursion: Windows

That it is necessary to specify the flag on Windows makes it seem to most people that Windows cannot delete open files, but that's actually not true. That's just the default behaviour.

CreateFile():

Enables subsequent open operations on a file or device to request delete access.

Otherwise, other processes cannot open the file or device if they request delete access.

If this flag is not specified, but the file or device has been opened for delete access, the function fails. Note Delete access allows both delete and rename operations.

DeleteFile():

The DeleteFile function marks a file for deletion on close. Therefore, the file deletion does not occur until the last handle to the file is closed. Subsequent calls to CreateFile to open the file fail with ERROR_ACCESS_DENIED.

Having an open handle to a file with no name is one of the most typical methods of creating unnamed temporary files: Create a new file, open it, delete the file. You now have a handle to a file that nobody else can open. On Unix, the filename is truly gone and on Windows you cannot open a file with the same name.

The question is now:

Does Files.newOutputStream() set FILE_SHARE_DELETE?

Looking at the source, you can see that shareDelete indeed defaults to true. The only way to reset it is to use the non-standard ExtendedOpenOption NOSHARE_DELETE.

So yes, you can delete opened files in Java unless they are explicitly locked.

Why can't I re-create the deleted file?

The answer to that is hidden in the documentation of DeleteFile() above: The file is only marked for deletion, the file is still there. On Windows, you cannot create a file with the name of a file marked for deletion until the file is properly deleted, i.e. all handles to the file are closed.

The possible confusion of mixing name deletion and actual file deletion is probably why Windows disallows deleting open files by default in the first place.

Why does Files.exists() return false?

Files.exists() in the deep end on Windows opens that file at some point and we already know that we cannot re-open a deleted-but-still-open file on Windows.

In detail: Java code calls FileSystemProvider.checkAccess()) with no arguments, which calls WindowsFileSystemProvider.checkReadAccess() which straight away tries to open the file and hence fails. From what I can tell, this is the path taken when you call Files.exist().

There is also another code path that calls GetFileAttributeEx() to retrieve file attributes. Once again, it is not documented what happens when you try to retrieve the attributes of an deleted but not yet removed file, but indeed, you cannot retrieve the file attributes of a file marked for deletion.

Guessing, I'd say that GetFileAttributeEx() calls GetFileInformationByHandle() at some point, which it will never get to because it cannot get a file handle in the first place.

So indeed, after DeleteFile() the file is gone for most practical purposes. It still has a name, however, shows up in directory listings and you cannot open a file with the same name until the original file had all its handles closed.

This behaviour is more or less consistent, because using GetFileAttributes() to check if a file exists is a actually an file accessibility check, which is interpreted as file exists. FindFirstFile() (used by Windows Explorer to determine the file list) finds file names but tells you nothing about accessibility of the names.

Welcome to a few more weird loops in your head.

Androgen

I'm not convinced by your arguments about the Windows implementation here and its similarity to linux. Your manual reference is about permissions on open not whether the actual delete operation will succeed. I'd need to see a low level example (written in C or similar) to prove the case. Usually windows has a much stronger link between the name and the file than POSIX systems. Files don't generally exist on windows if the name does not. – Blackmun

@couling Before I prove a teapot, what do you think FILE_SHARE_DELETE is for, then? – Androgen

its the fault of the manual for being unclear. Sorry I'm being pedantic, but what it states is Otherwise, other processes cannot open the file or device if they request delete access. This is talking about an "open" function not a "delete" function and does not explicitly say that deletion can occur while the file is open elsewhere. – Blackmun

Okay looking elsewhere, you're correct :D I never did like Microsoft documentation. – Blackmun

@couling Documentation for DeleteFile isn't any better, because it tells that "The DeleteFile function marks a file for deletion on close. Therefore, the file deletion does not occur until the last handle to the file is closed.". That does not answer the question if the file name is gone immediately. They might leave that to the filesystem implementation, which is probably why the API documentation is so vague. – Androgen

Actually it does. Well done on digging that up... So the file is not "deleted" until the last handle is closed... well the OP's code will keep a handle open (from the same program no less). The delete will succeed, because there is no error in doing so, but the file will not be deleted until the OutputStream is closed because of the open handle. The fairly accurately explains the behaviour. My understanding is that unlike Posix, in windows deleting a file does not delete the name until the content is gone. – Blackmun

Well, I've wrapped this up in an edit now. Thanks for the hints and the discussion. – Androgen

Upvoted; a nice and thorough answer. However, it's missing one part of the question: "Why does the following call to Files.exist() return false?" – Hawaii

@Hawaii Well, let me add that :-) – Androgen

@Hawaii This is all down to windows. File.exists() most likely is based on GetFileAttribtes which fails if the file is flagged for deletion or if the file doesn't exist. It seems java hits the error and just assumes it doesn't exist. msdn.microsoft.com/en-us/library/windows/desktop/… – Blackmun

Answer 2

If Files.delete did not throw exception it means it deleted the file. Files.delete javadoc says that "on some operating systems it may not be possible to remove a file when it is open and in use by this Java virtual machine or other programs".

Jephthah

But shouldn't it throw an IO / Access Exception then? - I mean now it's saying okay, deleted it, without actually deleting it. The deletion is happening the moment the outputstream is closed. So you could theoretically continue write content without knowing that a deletion will happen at the end... – Underlinen

@Underlinen Because the file name is gone, i.e. deletion was successful. However, the file still exists (until you close the last handle to it). Windows apps often make you believe that you cannot delete open files, but that's actually not true. – Androgen