After a lot of reading around, I came up with an answer.
A pretty good guide is here: http://nelenkov.blogspot.no/2011/12/using-custom-certificate-trust-store-on.html
Now, since I'm not using HTTPS, I had to come up with a slightly different approach for getting a clean SSL socket with the new keystore:
KeyStore store = KeyStore.getInstance("BKS");
InputStream truststore = mainActivity.getResources().openRawResource(R.raw.trust);
store.load(truststore, "PASSWORD".toCharArray());
TrustManagerFactory tmf = TrustManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
tmf.init(store);
SSLContext context = SSLContext.getInstance("TLS");
context.init(null, tmf.getTrustManagers(), new SecureRandom());
Socket socket = context.getSocketFactory().createSocket(ip, port);