How to know if the filesystem supports sparse file programmatically

Question

Java NIO provides us the option to create sparse files

    val path = Path("""E:\a""")

    val fileChannel = FileChannel.open(
        path,
        StandardOpenOption.READ,
        StandardOpenOption.WRITE,
        StandardOpenOption.CREATE_NEW,
        StandardOpenOption.SPARSE,
    )

The documentation says sparse file will be created if the OS and filesystem support it, but if it doesn't it will create a regular file.

But I want to know if the OS+filesystem does or does not support SPARSE so that, I can disable a feature from my app.

Whereby

Seems to be Windows. Does this help https://mcmap.net/q/1914717/-creating-a-sparse-file-in-java-on-ntfs/18157 ? – Delegacy

@JimGarrison I'd appreciate it if you could reread the question. – Whereby

This certainly seems like an XY problem. What problem does knowing if a filesystem supports sparse files purport to solve? – Rocher

@AndrewHenle I thought the OP's last paragraph was a clear statement of the problem. – Melone

@Melone And what problem does "so that, I can disable a feature from my app." solve? – Rocher

@AndrewHenle imagine you're a user and you see a UI with a tick box that says "enable sparse file functionality" but you have no idea what a sparse file is because your OS doesn't support them. As a developer, you don't want to show your users non-applicable settings. – Melone

I initially didn't wanna add because it felt off-topic but since some are interested let's explain why it could matter. – Whereby

Imagine you're making a multiconnection downloader where a file is written from different pointers. Now, if the filesystem doesn't support sparse and you start writing from the middle the half-size previous bytes are filled with null char. this may not seem like a problem but in reality, all NAND storage has a limited write cycle. If the filesystem doesn't support sparse and the user is a regular large file downloader, multi-connection might not be worth it for him. As a dev, I should warn them how multi-connection could affect them. – Whereby

Answer 1

If a file is a sparse file that means its physicalSize / logicalSize should be lover than 1. If you can calculate this values than you can undrestand if this file is sparse file or not.

To do that first you need to get the file is physicalSize. You should use different methods for windows and linux to calculate that value.

There is no direct API for that so you need to use ProcessBuilder for this:

for linux:

   Path path = Paths.get(filePath);
     File file = path.toFile();
    Long physicalSize = 0;
    // create and run the process
    Process process = new ProcessBuilder("du", "--block-size=1",file.getAbsolutePath()).redirectErrorStream(true).start();

        
        try (BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
            String line = reader.readLine();
            if (line != null) {
                String[] parts = line.split("\\s+");
                // take the value from return string  should be something like: 9379    filename
                physicalSize = Long.parseLong(parts[0]);
            }
        }

        process.waitFor();

After you capture the physical Size you need to capture the logical Size for the file, simply you can do it like this:

  Path path = Paths.get(filePath);
  long logicalSize = Files.size(path);

And finally you can use these 2 values to determine whether its a sparse file or not:

   double sparseness = (double) physicalSize / logicalSize;

            if (sparseness < 1) {
                System.out.println("a sparse file");
            } else {
                System.out.println("not a sparse file");
            }

Willable

The -b option is equivalent to --apparent-size --block-size=1 and that would output the logical, not physical, size. You don't want --apparent-size but just --block-size=1. – Melone

I've changed the "-b" to "--block-size=1" – Willable

Answer 2

Sparse files and filesystems

According to this page, only 3 major filesystems don't support sparse files:

• HFS+ (old MacOS FS)
• exFAT (used for USB flash drives)
• FAT32 (very old Windows FS)

Also note that sparse files are not created the same way on different filesystems. On NTFS, you need to explicitly make the file sparse. On other filesystems, files are automatically sparse (as demonstrated in this question).

How Java creates sparse files

On Windows, the StandardOpenOption.SPARSE flag is used in WindowsChannelFactory:

// make the file sparse if needed
if (dwCreationDisposition == CREATE_NEW && flags.sparse) {
    try {
        DeviceIoControlSetSparse(handle);
    } catch (WindowsException x) {
        // ignore as sparse option is hint
    }
}

The call will succeed in case of NTFS.

For Unix-based OSes, the flag is ignored in UnixChannelFactory:

case SPARSE : /* ignore */ break;

The file will be sparse for most filesystems except HFS+.

Conclusion

There is no pure Java solution to answer the question "does the filesystem support sparse files". However, in practice, only USB flash drives and old MacOS may not.

Patroclus

But if you try to create a sparse file on a USB stick formatted as FAT, irrespective of what OS you're on, it won't be sparse. – Melone

@Melone You're right, many USB sticks don't support sparse files. Added it to my conclusion. – Patroclus

If only there was an easy way to tell what file system a certain directory lives on. Even USB flash drives (which are usually FAT) can be formatted as NTFS or ext4. – Melone

Answer 3

You could create an empty file with size 10 KB, with sparse option, and check it's actual size on the filesystem. If it's actual size is less than 10 KB then sparse is supported. If it's 10KB - then it's not supported. And delete the file after the check.

Executant

"check it's actual size on the filesystem" How do you do that? – Patroclus

By using java.io.File length() – Executant

That gives the logical size. – Patroclus

You give it by controlling how many bytes you write to disk. E.g you write 10 kb all zeros - so the file logical size is 10kb, but if the sparse is enabled and supported on the fs, these zeros won't be written to disk, but described in a manifest, so the file actual size on the disc will be less than 10kb. – Executant

As Olivier said, File.length() gives the logical file size, not the physical size. That means it will return 10KB even if the file only occupies 1KB on the disk. The logical size is how many bytes your Java application can read from the file before it gets EOF. – Melone