What is a good way to deploy secret Java key stores in an OpenShift environment?

Question

We have a Java web application that is supposed to be moved from a regular deployment model (install on a server) into an OpenShift environment (deployment as docker container). Currently this application consumes a set of Java key stores (.jks files) for client certificates for communicating with third party web interfaces. We have one key store per interface.

These jks files get manually deployed on production machines and are occasionally updated when third-party certificates need to be updated. Our application has a setting with a path to the key store files and on startup it will read certificates from them and then use them to communicate with the third-party systems.

Now when moving to an OpenShift deployment, we have one docker image with the application that is going to be used for all environments (development, test and production). All configuration is given as environment variables. However we cannot give jks files as environment variables these need to be mounted into the docker container's file system.

As these certificates are a secret we don't want to bake them into the image. I scanned the OpenShift documentation for some clues on how to approach this and basically found two options: using Secrets or mounting a persistent volume claim (PVC).

Secrets don't seem to work for us as they are pretty much just key-value-pairs that you can mount as a file or have handed in as environment variables. They also have a size limit to them. Using a PVC would theoretically work, however we'd need to have some way to get the JKS files into that volume in the first place. A simple way would be to just start a shell container mounting the PVC and copying the files manually into it using the OpenShift command line tools, however I was hoping for a somewhat less manual solution.

Do you have found a clever solution to this or a similar problem where you needed to get files into a container?

Glottalized

Answer 1

It turns out that I misunderstood how secrets work. They are indeed key-values pairs that you can mount as files. The value can however be any base64 encoded binary that will be mapped as the file contents. So the solution is to first encode the contents of the JKS file to base64:

cat keystore.jks| base64

Then you can put this into your secret definition:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: my-namespace
data:
  keystore.jks: "<base 64 from previous command here>"

Finally you can mount this into your docker container by referencing it in the deployment configuration:

apiVersion: v1
kind: DeploymentConfig
spec:
  ...
  template:
    spec:
      ...
      container:
       - name: "my-container"
         ...
         volumeMounts:
            - name: secrets
              mountPath: /mnt/secrets
              readOnly: true

     volumes:
        - name: secrets
          secret:
            secretName: "my-secret"
            items:
              - key: keystore.jks
                path: keystore.jks

This will mount the secret volume secrets at /mnt/secrets and makes the entry with the name keystore.jks available as file keystore.jks under /mnt/secrets.

I'm not sure if this is really a good way of doing this, but it is at least working here.

Glottalized

How would you then use this keystore? By re-converting it into JKS format or something ? – Hairworm

The key store will be mounted into a folder in your container. E.g. in the example above it's mounted to /mnt/secrets/keystore.jks. From there you can load it using the standard keystore API (e.g. KeyStore ks = KeyStore.getInstance("JKS"); InputStream readStream = new FileInputStream("/mnt/secrets/keystore.jks"); ks.load(readStream, keystorePasswordCharArray);. – Crumple

One can link a secret to a service account so that the contents are injected into containers as files. See here - docs.openshift.com/container-platform/3.11/dev_guide/… But, the documentation does not clarify the path where the secrets get mounted. Any thoughts? – Apace

you can also directly create from file with kubectl create secret generic my-secret --from-file=keystore.jks=/path/to/file/keystore.jks – Cultrate

Answer 2

You can add and mount the secrets like stated by Jan Thomä, but it's easier like this, using the oc commandline tool:

./oc create secret generic crnews-keystore --from-file=keystore.jks=$HOME/git/crnews-service/src/main/resources/keystore.jks --from-file=truststore.jks=$HOME/git/crnews-service/src/main/resources/truststore.jks --type=opaque

This can then be added via UI: Applications->Deployments->-> "Add config files" where you can choose what secret you want to mount where.

Note, that the name=value pairs (e.g. truststore.jks=) will be used like filename=base64decoded-Content.

Imply

Answer 3

My generated base64 was multiline and I was getting the same error.

Trick is, use -w0 argument in base64 so that the whole encode is in 1 line!

base64 -w0 ssl_keystore.jks > test

Above will create a file named test and will contain the base64 in one line, copy paste like this in a secret:

apiVersion: v1
kind: Secret
metadata:
  name: staging-ssl-keystore-jks
  namespace: staging-space
type: Opaque
data:
  keystore.jsk: your-base64-in-one-line

Rapt

I just surrounded the multi-line output as instructed above and it worked just fine. – Twitch

Answer 4

Building upon what both @Frischling and @Jan-Thomä said, and in agreement with Frischling as his way was easier and took care of both the trust cert keystores, after adding the keystores as a secret, under Applications->Deployments->[your deployments name] Click the environment link and add the following system properties:
Name: JAVA_OPTS_APPEND and
Value -Djavax.net.ssl.keyStorePassword=changeme -Djavax.net.ssl.keyStore=/mnt/keystores/your_cert_key_store.jks -Djavax.net.ssl.trustStorePassword=changeme -Djavax.net.ssl.trustStore=/mnt/keystores/your_ca_key_store.jks

This effectively will as indicated, append the keystore file paths, passwords to the java options used by the application, for example JBoss/WildFly or Tomcat.

Landmark

Notable is that this solution might cause logging passwords in unencrypted state to console – Aggrieve

True, using a config map and System property injection might be better right? Though they could still see them if they logged into the shell, so maybe a config map is not good, but only other alternative is to hardcode them in the app right? – Landmark