How to connect to FTPS server with data connection using same TLS session?

Asked 4/9, 2015 at 13:1 Answered 1/12, 2023 at 17:35

Solved java ftp ftps apache-commons-net session-reuse

Environment: I'm using Sun Java JDK 1.8.0_60 on 64-bit Windows 7, using Spring Integration 4.1.6 (which internally appears to use Apache Commons Net 3.3 for FTPS access).

I'm attempting to integrate with our application an automatic download from our client's FTPS server. I've done so successfully with SFTP servers using Spring Integration without any trouble for other clients without issues, but this is the first time a client has required us to use FTPS, and getting it to connect has been very puzzling. While in my real application I'm configuring Spring Integration using XML beans, to try to understand what's not working I'm using the following test code (though I'm anonymizing the actual host/username/password here):

final DefaultFtpsSessionFactory sessionFactory = new DefaultFtpsSessionFactory();
sessionFactory.setHost("XXXXXXXXX");
sessionFactory.setPort(990);
sessionFactory.setUsername("XXXXXXX");
sessionFactory.setPassword("XXXXXXX");
sessionFactory.setClientMode(2);
sessionFactory.setFileType(2);
sessionFactory.setUseClientMode(true);
sessionFactory.setImplicit(true);
sessionFactory.setTrustManager(TrustManagerUtils.getAcceptAllTrustManager());
sessionFactory.setProt("P");
sessionFactory.setProtocol("TLSv1.2");
sessionFactory.setProtocols(new String[]{"TLSv1.2"});
sessionFactory.setSessionCreation(true);
sessionFactory.setCipherSuites(new String[]{"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"});

final FtpSession session = sessionFactory.getSession();
//try {
    final FTPFile[] ftpFiles = session.list("/");
    logger.debug("FtpFiles: {}", (Object[]) ftpFiles);
//} catch (Exception ignored ) {}
session.close();

I'm running this code with -Djavax.net.debug=all to get all the TLS debugging information printed.

The main "control" connection to the FTPS server works fine, but when it tries to open the data connection for the list (or any other data connection I've tried), I get a javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake, caused by java.io.EOFException: SSL peer shut down incorrectly. If I uncomment the swallowing-exceptions catch block around the session.list command, then I can see (though the javax.net.debug output) that the server sent the following message after rejecting the data connection SSL handshake:

main, READ: TLSv1.2 Application Data, length = 129
Padded plaintext after DECRYPTION:  len = 105
0000: 34 35 30 20 54 4C 53 20   73 65 73 73 69 6F 6E 20  450 TLS session 
0010: 6F 66 20 64 61 74 61 20   63 6F 6E 6E 65 63 74 69  of data connecti
0020: 6F 6E 20 68 61 73 20 6E   6F 74 20 72 65 73 75 6D  on has not resum
0030: 65 64 20 6F 72 20 74 68   65 20 73 65 73 73 69 6F  ed or the sessio
0040: 6E 20 64 6F 65 73 20 6E   6F 74 20 6D 61 74 63 68  n does not match
0050: 20 74 68 65 20 63 6F 6E   74 72 6F 6C 20 63 6F 6E   the control con
0060: 6E 65 63 74 69 6F 6E 0D   0A                       nection..

What appears to be happening (and this is my first time dealing with FTPS, though I've dealt with plain FTP before) is that the way the server ensures authentication and encryption over both the control and data connections is that after a "normal" TLS connection to establish the control connection and authentication happens there, each data connection requires the client to connect with the same TLS session. This makes sense to me as how it's supposed to be work, but the Apache Commons Net FTPS implementation doesn't seem to be doing that. It seems to be trying to establish a new TLS session, and so the server is rejecting the attempt.

Based on this question about resuming SSL sessions in JSSE, it appears that Java assumes or requires a different session for each host/post combination. My hypothesis is that since the FTPS data connection is on a different port than the control connection, it's not finding the existing session and is trying to establish a new one, so the connection fails.

I see three main possibilities:

The server is not following the FTPS standard in requiring the same TLS session on the data port as on the control port. I can connect to the server fine (using the same host/user/password as I'm trying to use in my code) using FileZilla 3.13.1. The server identifies itself as "FileZilla Server 0.9.53 beta" upon login, so perhaps this is some sort of proprietary FileZilla way of doing things, and there's something odd I need to do to convince Java to use the same TLS session.
The Apache Commons Net client doesn't actually follow the FTPS standard, and only allows some subset that doesn't allow for securing the data connections. This would seem odd, as it appears to be the standard way of connecting to FTPS from within Java.
I'm completely missing something and misdiagnosing this.

I'd appreciate any direction you can provide as to how to connect to this kind of FTPS server. Thank you.

Backsight answered 4/9, 2015 at 13:1 Comment(1)

Please refer this link for details #46631815 – Hertz 5/2, 2018 at 6:35

Indeed some FTP(S) servers do require that the TLS/SSL session is reused for the data connection. This is a security measure by which the server can verify that the data connection is used by the same client as the control connection.

Some references for common FTP servers:

vsftpd: https://scarybeastsecurity.blogspot.com/2009/02/vsftpd-210-released.html
FileZilla server: https://svn.filezilla-project.org/filezilla?view=revision&revision=6661
ProFTPD: http://www.proftpd.org/docs/contrib/mod_tls.html#TLSOptions (NoSessionReuseRequired directive)

What may help you with the implementation is that Cyberduck FTP(S) client does support TLS/SSL session reuse and it uses Apache Commons Net library:

https://github.com/iterate-ch/cyberduck/issues/5087 – Reuse Session key on data connection

See its FTPClient.java code (extends Commons Net FTPSClient), particularly its override of _prepareDataSocket_ method:

 @Override
 protected void _prepareDataSocket_(final Socket socket) {
     if(preferences.getBoolean("ftp.tls.session.requirereuse")) {
         if(socket instanceof SSLSocket) {
             // Control socket is SSL
             final SSLSession session = ((SSLSocket) _socket_).getSession();
             if(session.isValid()) {
                 final SSLSessionContext context = session.getSessionContext();
                 context.setSessionCacheSize(preferences.getInteger("ftp.ssl.session.cache.size"));
                 try {
                     final Field sessionHostPortCache = context.getClass().getDeclaredField("sessionHostPortCache");
                     sessionHostPortCache.setAccessible(true);
                     final Object cache = sessionHostPortCache.get(context);
                     final Method putMethod = cache.getClass().getDeclaredMethod("put", Object.class, Object.class);
                     putMethod.setAccessible(true);
                     Method getHostMethod;
                     try {
                         getHostMethod = socket.getClass().getMethod("getPeerHost");
                     }
                     catch(NoSuchMethodException e) {
                         // Running in IKVM
                         getHostMethod = socket.getClass().getDeclaredMethod("getHost");
                     }
                     getHostMethod.setAccessible(true);
                     Object peerHost = getHostMethod.invoke(socket);
                     putMethod.invoke(cache, String.format("%s:%s", peerHost, socket.getPort()).toLowerCase(Locale.ROOT), session);
                 }
                 catch(NoSuchFieldException e) {
                     // Not running in expected JRE
                     log.warn("No field sessionHostPortCache in SSLSessionContext", e);
                 }
                 catch(Exception e) {
                     // Not running in expected JRE
                     log.warn(e.getMessage());
                 }
             }
             else {
                 log.warn(String.format("SSL session %s for socket %s is not rejoinable", session, socket));
             }
         }
     }
 }

It seems that the _prepareDataSocket_ method was added to Commons Net FTPSClient specifically to allow the TLS/SSL session reuse implementation:
https://issues.apache.org/jira/browse/NET-426

A native support for the reuse is still pending:
https://issues.apache.org/jira/browse/NET-408
You will obviously need to override the Spring Integration DefaultFtpsSessionFactory.createClientInstance() to return your custom FTPSClient implementation with the session reuse support.

The above solution does not work on its own anymore since JDK 8u161.

According to JDK 8u161 Update Release Notes (and the answer by @Laurent):

Added TLS session hash and extended master secret extension support
...
In case of compatibility issues, an application may disable negotiation of this extension by setting the System Property jdk.tls.useExtendedMasterSecret to false in the JDK

I.e., you can call this to fix the problem (you still need to override the _prepareDataSocket_):

System.setProperty("jdk.tls.useExtendedMasterSecret", "false");

Though this should be a considered a workaround only. I do not know a proper solution.

An alternative implementation is here:
https://issues.apache.org/jira/browse/NET-408

There's a separate question about problems in 1.8.0_161:
SSL Session reuse in Apache FTPS client in JDK 8u161

I actually had the same problem in the past (just in C++/OpenSSL, I do not do Java), so I knew what to google for.

Kyliekylila answered 4/9, 2015 at 18:27 Comment(0)

You can use this SSLSessionReuseFTPSClient class :

import java.io.IOException;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.net.Socket;
import java.util.Locale;

import javax.net.ssl.SSLSession;
import javax.net.ssl.SSLSessionContext;
import javax.net.ssl.SSLSocket;

import org.apache.commons.net.ftp.FTPSClient;

public class SSLSessionReuseFTPSClient extends FTPSClient {

    // adapted from:
    // https://trac.cyberduck.io/browser/trunk/ftp/src/main/java/ch/cyberduck/core/ftp/FTPClient.java
    @Override
    protected void _prepareDataSocket_(final Socket socket) throws IOException {
        if (socket instanceof SSLSocket) {
            // Control socket is SSL
            final SSLSession session = ((SSLSocket) _socket_).getSession();
            if (session.isValid()) {
                final SSLSessionContext context = session.getSessionContext();
                try {
                    final Field sessionHostPortCache = context.getClass().getDeclaredField("sessionHostPortCache");
                    sessionHostPortCache.setAccessible(true);
                    final Object cache = sessionHostPortCache.get(context);
                    final Method method = cache.getClass().getDeclaredMethod("put", Object.class, Object.class);
                    method.setAccessible(true);
                    method.invoke(cache, String
                            .format("%s:%s", socket.getInetAddress().getHostName(), String.valueOf(socket.getPort()))
                            .toLowerCase(Locale.ROOT), session);
                    method.invoke(cache, String
                            .format("%s:%s", socket.getInetAddress().getHostAddress(), String.valueOf(socket.getPort()))
                            .toLowerCase(Locale.ROOT), session);
                } catch (NoSuchFieldException e) {
                    throw new IOException(e);
                } catch (Exception e) {
                    throw new IOException(e);
                }
            } else {
                throw new IOException("Invalid SSL Session");
            }
        }
    }
}

And With openJDK 1.8.0_161 :

We must set :

System.setProperty("jdk.tls.useExtendedMasterSecret", "false");

according to http://www.oracle.com/technetwork/java/javase/8u161-relnotes-4021379.html

Added TLS session hash and extended master secret extension support

In case of compatibility issues, an application may disable negotiation of this extension by setting the System Property jdk.tls.useExtendedMasterSecret to false in the JDK

Sexagesimal answered 6/3, 2018 at 14:4 Comment(5)

Setting this property works! Thank you for pointing this out! – Jonellejones 13/3, 2018 at 14:33

Can you explain me how to make it work ? I replace FTPSClient by SSLReuseSessionFTPSClient but it doesn't change anything – Breadboard 23/1, 2019 at 11:49

@Breadboard this should work, have you set the system property ?

SSLSessionReuseFTPSClient ftpsClient = null; 		try { 			ftpsClient = new SSLSessionReuseFTPSClient(); 			ftpsClient.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out), true)); 			...

– Sexagesimal 23/1, 2019 at 12:20

Thanks for your reply, I tried this : noelshack.com/2019-04-3-1548249527-ftpsclientproblem.png But it ended the same way as FTPSClient did. I Can't figure out how to make it works. – Breadboard 23/1, 2019 at 13:31

I added a breakpoint into prepareDataSocket method but it seems like my code doesn't go into it – Breadboard 23/1, 2019 at 13:49

If you can afford adding Bouncy Castle TLS org.bouncycastle:bctls-jdk18on as a dependency then you can try the following solution that avoids reflection and setting global properties:

class FTPSClientWithResumeBC extends FTPSClient {
    static {
        Security.addProvider(new BouncyCastleJsseProvider());
    }

    public FTPSClientWithResumeBC() {
        super(createSslContext());
    }

    private static SSLContext createSslContext() {
        try {
            // doesn't work with TLSv1.3
            SSLContext sslContext = SSLContext.getInstance("TLSv1.2", "BCJSSE");
            sslContext.init(null, new TrustManager[]{TrustManagerUtils.getValidateServerCertificateTrustManager()}, new SecureRandom()); // 1
            return sslContext;
        } catch (NoSuchAlgorithmException | NoSuchProviderException | KeyManagementException e) {
            throw new RuntimeException("Cannot create SSL context.", e);
        }
    }

    @Override
    protected void _connectAction_() throws IOException {
        super._connectAction_();
        execPBSZ(0);
        execPROT("P");
    }

    @Override
    protected void _prepareDataSocket_(Socket dataSocket) {
        if (_socket_ instanceof BCSSLSocket sslSocket) {
            BCExtendedSSLSession bcSession = sslSocket.getBCSession();
            if (bcSession != null && bcSession.isValid() && dataSocket instanceof BCSSLSocket dataSslSocket) {
                dataSslSocket.setBCSessionToResume(bcSession); // 2
                dataSslSocket.setHost(bcSession.getPeerHost()); // 3
            }
        }
    }
}

Notes:

worked for me™ with org.bouncycastle:bctls-jdk18on:1.77, commons-net:commons-net:3.10.0 and FileZilla Server 1.7.3
BouncyCastle TLS implementation has support for resuming a session - used on line marked with 2
line marked with 3 was required for one of my testing servers, it required SNI to be present in the ClientHello of the data connection (and FTPSClient did not do that by itself)
initialization on line marked 1 is similar to what FTPSClient does by default, you may need to adjust it to suit your needs
I am no expert on SSL nor FTPS so take the answer with some caution

Seizure answered 1/12, 2023 at 17:35 Comment(2)

This solution was most useful when you don't want to change the SSL/TLS behaviour for the rest of your application. There's even a version of the SSLContent.getInstance that can take a Provider object if you prefer not to declare BouncyCastle to the JVM-wide security manager for your app. Pay attention to the TLS 1.2 requirement - as currently TLS 1.3 session resumption is very different and might not be supported in the current v1.77 BouncyCastle. – Gaudery 8/3 at 6:42

Thanks a lot, It works for me but added bctls-jdk18on dependency to get BouncyCastleJsseProvider class – Materialize 2/5 at 15:59

To make Martin Prikryl's suggestion work for me I had to store the key not only under socket.getInetAddress().getHostName() but also under socket.getInetAddress().getHostAddress(). (Solution stolen from here.)

Durkee answered 25/9, 2016 at 19:23 Comment(0)

@Martin Prikryl's answer helped me.

According to my practice, it is worth mentioning that If you used

System.setProperty("jdk.tls.useExtendedMasterSecret", "false");

and it doesn't work, you can try JVM parameters of the same function:

-Djdk.tls.useExtendedMasterSecret=false.

Hope to help you.

Dibble answered 16/9, 2020 at 3:31 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Recommended topics

Hot tags