Apache Commons Net FTP is uploading corrupted files

Question

I'm trying to use Apache Commons Net for FTP file transfers.

Problem is files are intermittently arriving at the server corrupt. By 'corrupt' I mean that WinRAR tells me a ZIP file has an 'Unexpected end of archive'. Sometimes the files are completely empty. I have noticed that this happens more for larger files (100kb+), however does happen for small files too (20kb).

I know for a fact that the source zip file being uploaded is valid, and is only 243kb.

I do not get any errors/exceptions from the code.

Here's the code being executed:

int CON_TIMEOUT = (int) TimeUnit.SECONDS.toMillis(20); // fail if can't connect within 20 seconds
int LIVE_TIMEOUT = (int) TimeUnit.MINUTES.toMillis(5); // allow up to 5 minutes for data transfers

FTPClient client = new FTPClient();
client.setConnectTimeout(CON_TIMEOUT);
client.setDataTimeout(LIVE_TIMEOUT);
client.connect(host);
client.setSoTimeout(LIVE_TIMEOUT);
client.login(user, pass);
client.changeWorkingDirectory(dir);
log("client ready");

File file = new File(filePath);
String name = new Date().getTime() + "-" + file.getName();

InputStream fis = null;
try
{
    fis = new FileInputStream(file);
    if (!client.storeFile(name, fis))
        throw new RuntimeException("store failed");
    log("store " + name + " complete");
}
finally
{
    IOUtils.closeQuietly(fis);
    try
    {
        client.logout();
        log("logout");
    }
    catch (Throwable e)
    {
        log("logout failed", e);
    }
    try
    {
        client.disconnect();
        log("disconnect");
    }
    catch (Throwable e)
    {
        log("disconnect failed", e);
    }
}

and some logs:

2010-08-10 21:32:38 client ready
2010-08-10 21:32:49 store 1281439958234-file.zip complete
2010-08-10 21:32:49 logout
2010-08-10 21:32:49 disconnect
2010-08-10 21:32:50 client ready
2010-08-10 21:33:00 store 1281439970968-file.zip complete
2010-08-10 21:33:00 logout
2010-08-10 21:33:00 disconnect
2010-08-10 21:33:02 client ready
2010-08-10 21:33:11 store 1281439982234-file.zip complete
2010-08-10 21:33:11 logout
2010-08-10 21:33:11 disconnect
2010-08-10 21:33:15 client ready
2010-08-10 21:33:25 store 1281439995890-file.zip complete
2010-08-10 21:33:26 logout
2010-08-10 21:33:26 disconnect
2010-08-10 21:33:27 client ready
2010-08-10 21:33:36 store 1281440007531-file.zip complete
2010-08-10 21:33:36 logout
2010-08-10 21:33:36 disconnect
2010-08-10 21:33:37 client ready
2010-08-10 21:33:48 store 1281440017843-file.zip complete
2010-08-10 21:33:48 logout
2010-08-10 21:33:48 disconnect
2010-08-10 21:33:49 client ready
2010-08-10 21:33:59 store 1281440029781-file.zip complete
2010-08-10 21:33:59 logout
2010-08-10 21:33:59 disconnect
2010-08-10 21:34:00 client ready
2010-08-10 21:34:09 store 1281440040812-file.zip complete
2010-08-10 21:34:09 logout
2010-08-10 21:34:09 disconnect
2010-08-10 21:34:10 client ready
2010-08-10 21:34:23 store 1281440050859-file.zip complete
2010-08-10 21:34:24 logout
2010-08-10 21:34:24 disconnect
2010-08-10 21:34:25 client ready
2010-08-10 21:34:35 store 1281440065421-file.zip complete
2010-08-10 21:34:35 logout
2010-08-10 21:34:35 disconnect

Note that all of these were complete within 15 seconds, and all of the resulting files on the server are corrupt.

I have also tested without setting any timeouts and the problem still occurs.

Duiker

Answer 1

Commons FTP defaults to Ascii file types. You want to set it to Binary when dealing with binary data like a ZIP file.

From http://commons.apache.org/net/api/org/apache/commons/net/ftp/FTPClient.html

The default settings for FTPClient are for it to use FTP.ASCII_FILE_TYPE , FTP.NON_PRINT_TEXT_FORMAT , FTP.STREAM_TRANSFER_MODE , and FTP.FILE_STRUCTURE . The only file types directly supported are FTP.ASCII_FILE_TYPE and FTP.BINARY_FILE_TYPE .

You want to do setFileType(FTP.BINARY_FILE_TYPE) before you send the file.

Heflin

make sure to ftpClient.setFileType(FTP.BINARY_FILE_TYPE) after any connect method, since connect() will reset type to FTP.ASCII_FILE_TYPE. – Concretion

Not only after connect, it must go after login, see Apache Java FTP client does not switch to binary transfer mode on some servers. – Archeozoic

Answer 2

Solution

I had the same issue and solved it by calling

ftpClient.setFileType(FTP.BINARY_FILE_TYPE)

before each method retrieveFile, retrieveFileStream, storeFile

Explanation

File is corrupted, because default fileType is FTP.ASCII_FILE_TYPE. This causes the issue. If you are on linux all bytes \n\r (windows end of file) are changed into \n byte. And this corrupt the file.

To avoid this behavior you have to call ftpClient.setFileType(FTP.BINARY_FILE_TYPE). Unfortunately, this setup is reset by each connect method back to ASCII_FILE_TYPE. In my case this was reset even by method listFiles. I guess, that this happened because I use passiveMode on ftpClient.

So if you want to avoid troubles call setFileType(FTP.BINARY_FILE_TYPE) right before every file transfer.

Bootle

BTW: Sorry, but I have to say it: This behavior of 'Apache Commons Net FTP' is truly awful. – Bootle

The mode (at least with some FTP servers) resets with login, so setFileType has to be called after it. In general, it does not reset with listFiles, though it's possible that your particular FTP server does. The Apache Commons Net library itself never resets the mode on its own. – Archeozoic

Thanks - Good point - I remember, that call setFileType after login wasn't enough in my case. And you are right, that this issue is probably caused by the server and not by client library. – Bootle

Answer 3

I had this problem despite specifying binary file type so I wrote code to validate the uploaded file via MD5 hashing:

public void upload(String sourceFilePath) throws Exception
{
    while (true)
    {
        // Upload
        File sourceFile = new File(sourceFilePath);
        String sourceFileHash = MD5Checksum.getMD5Checksum(sourceFilePath);
        String remoteFile = sourceFile.getName();

        try (InputStream inputStream = new FileInputStream(sourceFile))
        {
            boolean successful = ftpClient.storeFile(remoteFile, inputStream);

            if (!successful)
            {
                throw new IllegalStateException("Upload of " + sourceFilePath + " failed!");
            }
        }

        // Download
        File temporaryFile = File.createTempFile("prefix", "suffix");
        try (OutputStream outputStream = new BufferedOutputStream(new FileOutputStream(temporaryFile)))
        {
            boolean successful = ftpClient.retrieveFile(remoteFile, outputStream);

            if (!successful)
            {
                throw new IllegalStateException("Download of " + sourceFilePath + " failed!");
            }
        }

        String downloadFileHash = MD5Checksum.getMD5Checksum(temporaryFile.getAbsolutePath());
        Files.delete(temporaryFile.toPath());

        // Make sure the file hashes match
        if (sourceFileHash.equals(downloadFileHash))
        {
            break;
        }
    }
}

MD5Checksum.java:

import java.io.*;
import java.security.MessageDigest;

public class MD5Checksum
{
    private static byte[] createChecksum(String filename) throws Exception
    {
        try (InputStream fileInputStream = new FileInputStream(filename))
        {
            byte[] buffer = new byte[1024];
            MessageDigest complete = MessageDigest.getInstance("MD5");
            int numRead;

            do
            {
                numRead = fileInputStream.read(buffer);
                if (numRead > 0)
                {
                    complete.update(buffer, 0, numRead);
                }
            } while (numRead != -1);

            return complete.digest();
        }
    }

    public static String getMD5Checksum(String filename) throws Exception
    {
        byte[] checksum = createChecksum(filename);
        StringBuilder result = new StringBuilder();

        for (byte singleByte : checksum)
        {
            result.append(Integer.toString((singleByte & 0xff) + 0x100, 16).substring(1));
        }

        return result.toString();
    }
}

The MD5 code is taken from here.

Arson