FileUtils.copyFile fails for large files

Question

I have a simple Java app that is trying to copy a file across the WAN (from Ireland to NY).

I recently modified it to use FileUtils because the native Java file copy was too slow. I researched and found that because Fileutils uses NIO it is better. The file copy now works great but occasionally I need to copy very large files (> 200Mb) and the copy fails with the error:

java.io.IOException: Failed to copy full contents from...

I know the error means that the destination file size is not the same as the source, so initially I figured it was network problems. The process tries repeatedly to copy the file every couple of hours but it is never successful. However, when I copy the file manually through a Windows exploer then it works fine. This would seem to rule out the network...but I'm not really sure.

I have searched but could not find any posts with the exact same issue. Any help would be greatly appreciated.

Thanks!

Addition:
I am using this FileUtils method:

public static void copyFile(java.io.File srcFile, java.io.File destFile) throws java.io.IOException

Dongola

There's a number of methods called copyFile in FileUtils. If you add some of your code to show exactly what you are doing we could provide more help. – Sheathbill

Possible duplicate #15805803 – Sheathbill

Hi, I saw the post you mentioned but that has to do with file locking. Each time I copy the file it has a unique name so I do not have that problem. – Dongola

Is the file possibly changing as you are copying it? What version of Apache IO are you using? – Sheathbill

Hi, I am using Commons IO 2.4. No, the file does not changes once it gets created. The app creates the file, names it with a time stamp, then tries to copy it. It runs every 2 hours so the next time it will have a new file name. Also note, smaller files do not have a problem. This only occurs occasionally when the process happens to create a large file. Thanks! – Dongola

If you are using Java 7 the advice seems to be to use NIO. #107270 – Sheathbill

I'm actaully using Java 6 but I will try upgrading and testing that in my QA environment. I'll let you know how it goes. Thanks! – Dongola

Answer 1

So I found the issue to be on the destination folder. There is a polling process that is suppose to pick up the file after it gets copied. However, the file was getting moved prior to the copy being completed. This probably wouldn't happen on a windows drive because the file would be locked (i tested locally and i could not delete while file is copying). However, the destination folder is a mounted a celerra share. The unix process under the hood is what grabs the file...I guess it doesn't care if some windows process is still writing to it.

Thanks for your time medPhys-pl!

Dongola

I realise this question is now more than 6 years old, but I am curios as to what you did- change the poller behaviour (which seems unlikely) or what? – Undulation

I forgot about this one! Yes, I ended up dropping a separate file as a flag and the polling process waits for it before sweeping the directory. – Dongola

ah. Unfortunately, in my case I o not have ownership of the poller. Too bad. – Undulation