How copy files bigger than 4.3 GB in java

Question

I'm writing a program part simply to copy a file from source to a destination File. The code works as well it should but if there is a file large file the process of copy will end up, after the destination file reach a size of 4.3 GB, with an exception. The exception is a "file is to large" it looks like:

java.io.IOException: Die Datei ist zu groß
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:101)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at java.nio.file.Files.copy(Files.java:2909)
at java.nio.file.Files.copy(Files.java:3069)
at sample.Controller.copyStream(Controller.java:318)

The method to produce that is following:

    private void copyStream(File src, File dest){
    try {

        FileInputStream fis = new FileInputStream(src);
        OutputStream newFos = java.nio.file.Files.newOutputStream(dest.toPath(),StandardOpenOption.WRITE);

        Files.copy(src.toPath(),newFos);

        newFos.flush();
        newFos.close();
        fis.close();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

I also tried to use java.io Fileoutputstream and write in a kbyte way, but there happends the same. How can I copy or create files larger than 4.3 GB? Is it maybe possible in other language than java? This programm I run on a Linux (Ubuntu LTS 16.04).

Thanks in advance.

---

Edit:

Thanks very much you all for your help. As you said, the file system was the problem. After i formated the file system to exfat it works fine.

Mulch

What kind of filesystem are you writing to? FAT32 aka VFAT does not support files over 4.3GB, no matter what tool or language you use. – Mymya

ok, then that may be the problem the extern hdd is using VFAT. Is there a better recomidation of file system? – Mulch

If you don't need to share it with Windows, ext4 is the standard choice. If you do need to share it with Windows, maybe exFAT or NTFS. – Mymya

This is an error which comes from the OS, you need another filesystem. – Rayleigh

Ok, thanks. Im trying now with an other file system. Then tell xou the succes. – Mulch

Also see Standard concise way to copy a file in Java?, File copy/move methods and approaches explanation, comparison, Reading and writting a large file using Java NIO, etc. – Anasarca

Answer 1

POSIX (and thus Unix) systems are allowed to impose a maximum length on the path (what you get from File.getPath() or the components of a path (the last of which you can get with File.getName()). You might be seeing this problem because of the long name for the file.

In that case, the file open operating system call will fail with an ENAMETOOLONG error code.

However, the message "File too large" is typically associated with the ´EFBIG´ error code. That is more likely to result from a write system call:

An attempt was made to write a file that exceeds the implementation-dependent maximum file size or the process' file size limit.

Perhaps the file is being opened for appending, and the implied lseek to the end of the file is giving the EFBIG error.

In the end, you could try other methods of copying if it has to do something with your RAM.

Also another option could be that the disk is full.

To copy files there are basically four ways [and it turns out streams is the fastest on a basic level] :

Copy with streams:

private static void copyFileUsingStream(File source, File dest) throws IOException {
    InputStream is = null;
    OutputStream os = null;
    try {
        is = new FileInputStream(source);
        os = new FileOutputStream(dest);
        byte[] buffer = new byte[1024];
        int length;
        while ((length = is.read(buffer)) > 0) {
            os.write(buffer, 0, length);
        }
    } finally {
      is.close();
      os.close();
    }
}

Copy with Java NIO classes:

private static void copyFileUsingChannel(File source, File dest) throws IOException {
    FileChannel sourceChannel = null;
    FileChannel destChannel = null;
    try {
        sourceChannel = new FileInputStream(source).getChannel();
        destChannel = new FileOutputStream(dest).getChannel();
        destChannel.transferFrom(sourceChannel, 0, sourceChannel.size());
    }finally{
           sourceChannel.close();
           destChannel.close();
    }
}

Copy with Apache Commons IO FileUtils:

private static void copyFileUsingApacheCommonsIO(File source, File dest) throws IOException {
    FileUtils.copyFile(source, dest);
}

and your Method by using Java 7 and the Files class:

private static void copyFileUsingJava7Files(File source, File dest) throws IOException {
    Files.copy(source.toPath(), dest.toPath());
}

Edit 1: as suggested in the comments, here are three SO-questions, which cover the problem and explain the four different methodes of copying better:

• Standard concise way to copy a file in Java?
• File copy/move methods and approaches explanation, comparison
• Reading and writing a large file using Java NIO

Thanks to @jww for pointing it out

Shwalb

ok thanks. The file name and the path length are not the problem i tried out, its just the file size. I will try a other filesystem. – Mulch