Using Java Mail StartTLS with a Truststore

Question

I'm trying to connect to a mail server which uses StartTLS with a self signed certificate via Java mail API. And that seems to be a problem, because i can't find any way to set accepted certificates or a truststore for StartTLS.

Properties props = new Properties();
props.put("mail.imap.starttls.enable", "true");
props.put("mail.imap.starttls.required", "true");
Session session = Session.getInstance(props);
Store store = session.getStore("imap");
store.connect(hostName, port, userName, userPassword);

When i run my application as is, i get this PKIX path error:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I would prefer not to use VM parameters like "-Djavax.net.ssl.trustStore" because i want to be able to control trusted certificates per access.

Sidenote: I've seen people use "mail.imap.socketFactory.class" to set their own implementation of SocketFactory with a self defined TrustManager. But when i do that my connection fails with

javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

I think this is because setting the socket factory will actually use SMTP over SSL instead of StartTLS (which starts as a plain text connection and switches to TLS later).

Funds

Answer 1

I have this working for an SMTP connection (not IMAP) using com.sun.mail:javax.mail:1.5.5 and (root) certificates loaded from a (not so standard) pfx-file. The properties given to Session.getInstance(props) are build in the following manner (see also the API-docs here and here, I think you can simply replace smtp with imap for most of the properties):

"mail.transport.protocol", "smtp"
"mail.smtp.host", "hostname"
"mail.smtp.port", "25"
"mail.smtp.connecttimeout", "5000" // 5 seconds
"mail.smtp.timeout", "50000" // 50 seconds
"mail.smtp.ssl.protocols", "TLSv1.2"
"mail.smtp.starttls.required", "true"

Now build a SSL Socket Factory using com.sun.mail.util.MailSSLSocketFactory (read the API-docs in the link):
MailSSLSocketFactory sslSocketFactory = new MailSSLSocketFactory("TLSv1.2");
Create and intialize a (default) KeyManagerFactory kmf (e.g. by loading a keystore).
Create and intialize a (default) TrustManagerFactory tmf.
Call sslSocketFactory.setKeyManagers(kmf.getKeyManagers()) and sslSocketFactory.setTrustManagers(tmf.getTrustManagers())
Set property "mail.smtp.ssl.socketFactory" to the sslSocketFactory instance (use the props.put(k,v)-method). Note that the socket factory instance that was created and configured is set, not some String or class. Javamail will use the set socket factory instance directly.
Use the properties to create a session.

Make sure you have logging configured properly and set it to TRACE for com.sun.mail. Logging shows exactly what is "going over the line" and in my case shows for example:
DEBUG com.sun.mail.smtp - Found extension "STARTTLS", arg ""
...
TRACE com.sun.mail.smtp.protocol - STARTTLS
TRACE com.sun.mail.smtp.protocol - 220 Ready to start TLS

On a side note: creating a default keystore and trustore can be done using this SslUtils class, methods loadKeyStore(null) and createDefaultTrustStore() (I created this utility class a while ago to help me load the not-so-standard pfx-files).

Chaudfroid

In addition to the above, you can do much of this simpler by just setting the mail.smtp.ssl.trust property to the name of the host you want to trust. Or, you can use the InstallCert program to load your self-signed certificate from the server into your default trust store. – Thistle

I was using a 1.3 release of Java mail and had to upgrade to get MailSSLSocketFactory. Works like a charm. – Funds