FileChannel.transferFrom fails for larger files with Out of memory Error

Question

FileChannel.transferFrom(source, 0, source.size()) gives the following OutOfMemory exception, when trying to copy a file of size around 2GB. I understand the memory issue due to larger files. Can we solve it by coping the file in small chunks, in a loop?

01-22 17:27:03.365: W/System.err(28538): java.io.IOException: mmap failed: ENOMEM (Out of memory)
01-22 17:27:03.375: W/System.err(28538):    at java.nio.MemoryBlock.mmap(MemoryBlock.java:119)
01-22 17:27:03.375: W/System.err(28538):    at java.nio.FileChannelImpl.map(FileChannelImpl.java:249)
01-22 17:27:03.380: W/System.err(28538):    at java.nio.FileChannelImpl.transferFrom(FileChannelImpl.java:381)
01-22 17:27:03.380: W/System.err(28538):    at com.druva.inSync.util.InSyncIOUtils.copyFile(InSyncIOUtils.java:123)
01-22 17:27:03.380: W/System.err(28538):    at com.druva.inSync.AsyncTasks.ProcessUploadTask.getFileItemForFile(ProcessUploadTask.java:102)
01-22 17:27:03.380: W/System.err(28538):    at com.druva.inSync.AsyncTasks.ProcessUploadTask.processUploads(ProcessUploadTask.java:124)
01-22 17:27:03.380: W/System.err(28538):    at com.druva.inSync.AsyncTasks.ProcessUploadTask.doInBackground(ProcessUploadTask.java:53)
01-22 17:27:03.380: W/System.err(28538):    at com.druva.inSync.AsyncTasks.ProcessUploadTask.doInBackground(ProcessUploadTask.java:1)
01-22 17:27:03.380: W/System.err(28538):    at android.os.AsyncTask$2.call(AsyncTask.java:287)
01-22 17:27:03.380: W/System.err(28538):    at java.util.concurrent.FutureTask.run(FutureTask.java:234)
01-22 17:27:03.380: W/System.err(28538):    at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:230)
01-22 17:27:03.380: W/System.err(28538):    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
01-22 17:27:03.380: W/System.err(28538):    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
01-22 17:27:03.380: W/System.err(28538):    at java.lang.Thread.run(Thread.java:841)
01-22 17:27:03.385: W/System.err(28538): Caused by: libcore.io.ErrnoException: mmap failed: ENOMEM (Out of memory)

Edit:

I tried the following code :

    for (long n = 0, s = source.size() >> 1; n < s;) {
     Log.d("copy file", "inside for loop " + destination.size()); 
     long c = destination.transferFrom(source, n, s - n); 
     n += c; 
     Log.d("copy file", "results: c=" + c + ", n=" + n); 
    } 

but it copies only the first half of the file...

Adcock

Just so you know why your code didn't work, here's an explanation: source.size()>>1 is essentially source.size()/2. Your loop basically says "try to transfer a block the size of half the source to the destination. Store how many bytes were actually transferred into c". Then it adds c to n - so you set n to half the size of the source if transferFrom worked. Your for loop then checks n<s, which is false, because you just set n=s=half the size of the source, so the for loop is done, and you've copied half the file. – Tartary

To continue my previous explanation: please note that if tranferFrom happened to fail in your example code, it would return zero. You then would add zero to n, and your for loop would check n<s which would be true because n didn't increase, and you would be in an infinite loop. – Tartary

Answer 1

I ran into the same issue with FileChannel.transferFrom, but I even occasionally saw the ENOMEM error with files less than 512MB.

Here's the code I ended up using to transfer the file in smaller blocks:

  // Transfer file in 256MB blocks
  final long blockSize = Math.min(268435456, sourceChannel.size());
  long position = 0;
  while (destinationChannel.transferFrom(sourceChannel, position, blockSize) > 0) {
    position += blockSize;
  }

The transferFrom method returns the number of bytes it succeeded in copying. If it returns a number greater than zero, it copied the whole block, so attempt to copy another block. When it doesn't copy anything, you're done.

Tartary

This worked for me, but I had to transfer in smaller block even with largeHeap=true. I used 50MB and it worked well, thanks! – Pulchia