Deleting a file based on disk ID

Question

As described here, using SetFileInformationByHandle with FILE_DISPOSITION_INFO allows one to set a file with an open handle to be deleted upon all handles being closed.

However, I am trying to delete a file based on its file index (disk ID) retrieved by FILE_DISPOSITION_INFO and OpenFileById in order to safely delete files/directories in a directory which differ only in case. This is safe to do in my use case, as on an NTFS system, file indexes are persistent until deletion, negating the use of ReplaceFile, which the current codebase handles.

However, when attempting to delete the handle, I get error 87 (ERROR_INVALID_PARAMETER). If I delete using a handle created with CreateFileW, I run into no problems. I can't do this, though, as Windows will not be able to distinguish between two file/folders of the same case, even though NTFS can.

I am also aware that there is an ambiguity with hardlinked files opened with OpenFileById, as hardlinked files share the same disk ID. The issue of hardlinked files can be considered irrelevant for this scenario. I will only be deleting directories by ID, which cannot be hardlinked.

Is there a parameter or setting I am missing in my OpenFileById call? Somehow, in my SetFileInformationByHandle call?

Additional methods I have tried:

• Calling DuplicateHandle with the OpenFileById handle, providing DELETE for dwDesiredAccess, and using that. Same ERROR_INVALID_PARAMETER result.
• Using ReOpenFile with the OpenFileById handle, providing DELETE for dwDesiredAccess, and using that. Same ERROR_INVALID_PARAMETER result.
• Using ReOpenFile with the OpenFileById handle, providing DELETE for dwDesiredAccess, and providing the FILE_FLAG_DELETE_ON_CLOSE flag. No error is given, but the file remains after all handles are closed.

Here is a minimal, yet complete, example which reproduces the problem:

#include <stdio.h>
#include <sys/stat.h>
#include <Windows.h>

DWORD getFileID(LPCWSTR path, LARGE_INTEGER *id)
{
    HANDLE h = CreateFileW(path, 0, 0, 0, OPEN_EXISTING,
        FILE_FLAG_OPEN_REPARSE_POINT |
        FILE_FLAG_BACKUP_SEMANTICS |
        FILE_FLAG_POSIX_SEMANTICS,
        0);
    if (h == INVALID_HANDLE_VALUE)
        return GetLastError();

    BY_HANDLE_FILE_INFORMATION info;
    if (!GetFileInformationByHandle(h, &info))
    {
        DWORD err = GetLastError();
        CloseHandle(h);
        return err;
    }
    id->HighPart = info.nFileIndexHigh;
    id->LowPart = info.nFileIndexLow;
    CloseHandle(h);
    return ERROR_SUCCESS;
}

DWORD deleteFileHandle(HANDLE fileHandle)
{
    FILE_DISPOSITION_INFO info;
    info.DeleteFileW = TRUE;
    if (!SetFileInformationByHandle(
        fileHandle, FileDispositionInfo, &info, sizeof(info)))
    {
        return GetLastError();
    }
    return ERROR_SUCCESS;
}

int wmain(DWORD argc, LPWSTR argv[])
{
    if (argc != 3)
    {
        fwprintf(stderr, L"Arguments: <rootpath> <path>\n");
        return 1;
    }

    DWORD err;
    HANDLE rootHandle = CreateFileW(
        argv[1], 0, 0, 0, OPEN_EXISTING,
        FILE_FLAG_OPEN_REPARSE_POINT |
        FILE_FLAG_BACKUP_SEMANTICS |
        FILE_FLAG_POSIX_SEMANTICS,
        0);
    if (rootHandle == INVALID_HANDLE_VALUE)
    {
        err = GetLastError();
        fwprintf(stderr,
            L"Could not open root directory '%s', error code %d\n",
            argv[1], err);
        return err;
    }

    LARGE_INTEGER fileID;
    err = getFileID(argv[2], &fileID);
    if (err != ERROR_SUCCESS)
    {
        fwprintf(stderr,
            L"Could not get file ID of file/directory '%s', error code %d\n",
            argv[2], err);
        CloseHandle(rootHandle);
        return err;
    }
    fwprintf(stdout,
        L"The file ID of '%s' is %lld\n",
        argv[2], fileID.QuadPart);

    FILE_ID_DESCRIPTOR idStruct;
    idStruct.Type = FileIdType;
    idStruct.FileId = fileID;
    HANDLE fileHandle = OpenFileById(
        rootHandle, &idStruct, DELETE, FILE_SHARE_DELETE, 0,
        FILE_FLAG_OPEN_REPARSE_POINT | FILE_FLAG_BACKUP_SEMANTICS);
    if (fileHandle == INVALID_HANDLE_VALUE)
    {
        err = GetLastError();
        CloseHandle(rootHandle);
        fwprintf(stderr,
            L"Could not open file by ID %lld, error code %d\n",
            fileID.QuadPart, err);
        return err;
    }

    err = deleteFileHandle(fileHandle);
    if (err != ERROR_SUCCESS)
    {
        fwprintf(stderr,
            L"Could not delete file by ID '%lld', error code %d\n",
            fileID.QuadPart, err);
    }

    CloseHandle(fileHandle);
    struct _stat _tmp;
    fwprintf(stdout,
        L"File was %ssuccessfully deleted\n",
        (_wstat(argv[2], &_tmp) == 0) ? L"not " : L"");
    CloseHandle(rootHandle);
    return err;
}

Any solution must work with Vista and above. Suggestions for code improvement are also welcome.

Kokaras

Try using DuplicateHandle on the handle you get from OpenFileById, with dwDesiredAccess set to DELETE. – Disproof

@HarryJohnston Good idea, unfortunately didn't work. I've edited the question to include what I've tried so far. – Kokaras

Presumably the system kernel is configured to be case sensitive, or you wouldn't have such files in the first place; I take it using NtCreateFile isn't an option? – Disproof

@HarryJohnston The files were created in Linux. Theoretically, the program should also be able to work with files with invalid names if it can delete by disk ID. – Kokaras

It doesn't look like it can, though. The problem presumably being that a disk ID identifies a file rather than a directory entry, and you can only delete a directory entry. Granted removing a directory by disk ID would be possible in principle since in NTFS a directory can't be hard linked, it doesn't look like Windows actually supports doing so. I think you'll need to set obcaseinsensitive to zero and use the kernel API. – Disproof

Alternatively, if you don't need to keep the conflicting names, it should be possible to rename them one at a time - presumably if you open a file by name when there are multiple matches you get a handle to one of them at random? – Disproof

@HarryJohnston Unfortunately, this will be used on machines where setting kernel settings and restarting would be impossible. As far as I can tell in my experimentation, opening a file by name when there are multiple matches opens the one with the first filename when sorted, so capitals first. It's confusing that using GetFinalPathNameByHandle gives the first hardlink path of a file. I have absolutely no experience in Windows kernel development. If I could use the POSIX subsystem simply, this would be easy. – Kokaras

I gather the POSIX subsystem no longer exists in the newest Windows releases, so no help there. There's still an NFS server so that's a potential option, but not available on all editions of Windows. What about renaming them so they don't conflict? If there are only two directories with matching names, and you want to remove one of them anyway, that should work perfectly - you don't need to be able to predict which one will wind up with which name, as you can check the file IDs. Another thought: could you mount the file system in question in a VM? – Disproof

@HarryJohnston The primary restriction is that is must work on Vista+ with default configurations, and be capable of running in the background. So, no NFS, no installations, and no reboots. A problem I see with renaming is that when a child of a directory is locked, the directory cannot be renamed (IIRC). The strange thing is I'd expect SetFileInformationByHandle to return ERROR_INVALID_HANDLE if the handle was incapable. It's also possible to find the first file path of a OpenFileById handle. Lastly, renaming simply wouldn't work for paths with "invalid" characters, like \ or *. – Kokaras

OK, so POSIX wouldn't have helped in the first place, never mind then. I'm not sure why a Linux directory would be in use while Windows has the file system mounted, but if it is you won't be able to rename it. You hadn't mentioned invalid characters before, I'm not sure you can do anything with such a file in Windows even via the kernel, I think you would have to dismount the file system and manipulate it directly. Given all your requirements, I think the answer is simply that this isn't possible. Someone else might have a clever idea though, so good luck. – Disproof

(Forgot to mention, ERROR_INVALID_HANDLE is typically only used if the handle you pass isn't actually a handle or is a handle to the completely wrong kind of object, e.g., calling SetEvent on a file handle. Using a handle of the right sort but that lacks the necessary properties typically produces ERROR_INVALID_PARAMETER or ERROR_ACCESS_DENIED. For example, calling DeleteFile on a directory produces ERROR_ACCESS_DENIED.) – Disproof

@HarryJohnston In a dual-booted system with Linux, mounting an NTFS partition opens it in "POSIX mode", where the only restrictions are that a path cannot contain NUL or /. A music manager that edits paths would add : to some song files. The paths allowed by Windows are a subset. Using a POSIX namespace (without installation) would fix this when files created in Linux are attempted to be deleted in Windows, but I don't see how. The peculiar thing is that you can use GetFinalPathNameByHandle and still get the name of the first hardlink of a file opened by ID, even the path is invalid. – Kokaras

Hmmm. It might (or might not) be possible to manipulate the files by passing the relevant control codes directly to the NTFS driver, bypassing the kernel's file system support. But it would probably have to be done from a device driver, not an application. (It might not need to be a kernel-mode device driver necessarily.) It would definitely be challenging for anyone without device driver (and preferable file system driver) experience. – Disproof

SWAG: As dumb as it sounds, might you not need to include POSIX_SEMANTICS on the OpenFileById call to get a "compatible" handle? Yeah - I know the POSIX_SEMANTICS nominally only has to do with file names...but there are dumber things in the world. @HarryJohnston mentioned the ERROR_INVALID_PARAMETER - maybe internally, it's "grossly" comparing flags. – Kore

If you're not too concerned with performance, you might try NTQueryInformationFile querying for FILE_NAME_INFORMATION, which takes the file handle and returns a full path to the file, which you can turn around and open with CreateFile(). In such a case, you might still want to use POSIX_SEMANTICS to ensure you're getting compatible behavior by the underlying subsystem when you open the file w/ OpenFileById() – Kore

@Kore OpenFileById does not take the FILE_FLAG_POSIX_SEMANTICS flag. Also, querying for the filename will still fail if the filename is invalid/differs only in case. – Kokaras

For the record, the documentation here: File System Behavior Overview (PDF) confirms (section 4.3.2) that you can't set the delete-on-close flag for a handle that was opened by ID. – Disproof

@HarryJohnston Thank you for (as far as I can tell) officially confirming this is unsupported by Microsoft. Is there any other option you know to guarantee the deletion of a file with incredibly unusual names? Is there everywhere-installed access to the POSIX subsystem hanging around somewhere I can't find? – Kokaras

I wouldn't necessarily know if there was, but it seems unlikely. Windows doesn't generally support that sort of edge case out of the box. – Disproof

Answer 1

There's a user mode version of the kernel mode ZwCreateFile called NTCreteFile which, among other things will give you all of the access rights you can't get with OpenFileById (but you can get with CreateFile). It can do everything CreateFile can do and more. For example, it can even create directories.

The good part is, there's an immensely hacky (but entertaining) way of specifying a file ID in the POBJECT_ATTRIBUTES argument as well, so you get the best of all worlds...except that it's an even more awkward API to call than your run-of-the-mill awkward Windows APIs.

There are two versions of the documentation. One at:

https://msdn.microsoft.com/en-us/library/bb432380(v=vs.85).aspx

and one at:

https://msdn.microsoft.com/en-us/library/windows/hardware/ff556465(v=vs.85).aspx

...which links to the ZwCreateFile documentation at:

https://msdn.microsoft.com/en-us/library/windows/hardware/ff566424(v=vs.85).aspx

The reason I point this out is that the first article omits some of the goodies (like opening files by ID) that are documented in the last article. I have found this to be common and have also found that most of the documented Zwxxx functionality actually does exists in the equivalent, but incompletely documented NTxxx functions. So you gotta hold your mouth just right to get the requisite functionality.

Kore

Answer 2

In order to make FILE_DISPOSITION_INFO work you need to specify the DELETE access in the CreateFile function as reported in https://msdn.microsoft.com/en-us/library/windows/desktop/aa365539(v=VS.85).aspx:

You must specify appropriate access flags when creating the file handle for use with SetFileInformationByHandle. For example, if the application is using FILE_DISPOSITION_INFO with the DeleteFile member set to TRUE, the file would need DELETE access requested in the call to the CreateFile function. To see an example of this, see the Example Code section. For more information about file permissions, see File Security and Access Rights. I.e.

//...
  HANDLE hFile = CreateFile( TEXT("tempfile"), 
                             GENERIC_READ | GENERIC_WRITE | DELETE,  //Specify DELETE access!
                             0 /* exclusive access */,
                             NULL, 
                             CREATE_ALWAYS,
                             0, 
                             NULL);

But it seems that an handle created with OpenFileById() cannot be used because the function cannot accept the DELETE flag.
From https://msdn.microsoft.com/en-us/library/windows/desktop/aa365432(v=vs.85).aspx on OpenFileById() it can be read: dwDesired

Access [in]
The access to the object. Access can be read, write, or both.

Even setting DELETE or GENERIC_ALL the function fails.
If you replace the handle passed to SetFileInformationByHandle with one created with the CreateFile function having the DELETE flag set, as above, it works.

Slap

CreateFile is not being used to create the handle that FILE_DISPOSITION_INFO is being used on, OpenFileById is. GENERIC_ALL was used for access, which includes delete. Replacing it with GENERIC_READ | GENERIC_WRITE | DELETE makes no difference. – Kokaras

I already know using CreateFile works, as described. This still has the problem of file paths not being unique with CreateFile. Is it possible to delete a file by disk ID in general, or reliably delete two files which differ only in case, if one is a file and the other a directory? – Kokaras

MS file system is not case sensitive, so in general the answer is no. I have checked what for me is the last chance: using ReOpenFile to reopen the flie created with OpenFileById adding the DELETE flag, but also this solution doesn't work :( – Slap

The problem is NTFS is case sensitive but Windows is not. Good idea on ReOpenFile, sad it didn't work. FindNextFile and similar methods still see them as separate files, however. – Kokaras

It seems that OpenFileById is crafted for internetwork files, so have a limited access to the real file. Maybe there is a function to change something on the opened file. Not one of the most used, nor one that I could recall. Maybe you want search in that direction something like ReOpenFile... – Slap

Answer 3

Have you looked into FILE_FLAG_POSIX_SEMANTICS? It will allow you to open files that differ only in case using CreateFile.

Edit: I guess I should have read your code first as I see you are using said flag.

Abadan

Doesn't work. Have you tried it yourself? Also fails with invalid names. – Kokaras

I'm trying your sample right now and I can reproduce the behavior. I guess my question to you now is which part do you want to get working, deleting files that differ only in case or deleting files opened by ID? – Abadan

Deleting files opened by ID, one benefit being able to delete files which differ only by case, as well as invalid filenames, such as those that contain :. FILE_FLAG_POSIX_SEMANTICS also does not work as described if the kernel is not currently case insensitive, which is the default. If you can form a code sample which you verify works, that might be useful. – Kokaras

Answer 4

Assume the files are XXX and xxx and you want to delete XXX.

1. MoveFile("XXX", "I think it's XXX")
2. If XXX got renamed, then DeleteFile("I think it's XXX")
3. Otherwise, DeleteFile("XXX"); MoveFile("I think it's XXX", "xxx")

As to OpenFileById, as you noted, there is a potential ambiguity with a file with multiple names (aka hard links). Allowing DELETE access could cause havoc with this, with an unexpected name being deleted (if it were left to the file system to select which one). I suspect they opted for the simple case of never letting DELETE access be granted.

A similar argument could be made for allowing hard links to directories. Sure, you could do it some of the time correctly, but once you created a cycle, things get a lot tougher...

Alack