Kubernetes Secret TLS Certificate P12 and Spring Boot Deployment doesn't work

Question

I'm currently stuck and don`t know how to proceed.

This is my Spring Boot application.properties

...
spring.datasource.driverClassName=org.postgresql.Driver
spring.datasource.url=jdbc:postgresql://${POSTGRES_HOST}:5432/postgres
spring.datasource.username=${POSTGRES_USER}
spring.datasource.password=${POSTGRES_PASSWORD}
spring.datasource.testWhileIdle=true
spring.datasource.validationQuery=SELECT 1
spring.jpa.show-sql=true
spring.jpa.hibernate.ddl-auto=update
spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.ImprovedNamingStrategy
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect

#Setup SSL
server.port: 8443
server.ssl.key-store: ${TLS_CERTIFICATE}
server.ssl.key-store-password: ${TLS_PASSWORD}
server.ssl.keyStoreType: PKCS12
server.ssl.keyAlias fundtr
...

My Deployment yaml for Spring Boot Application:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: f-app
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      name: f-app
      labels:
        app: f-app
    spec:
      containers:
      - name: f-app
        image: eu.gcr.io/..../...
        env:
          - name: POSTGRES_USER
            valueFrom:
              configMapKeyRef:
                name: postgres-config
                key: postgres_user
          - name: POSTGRES_PASSWORD
            valueFrom:
              configMapKeyRef:
                name: postgres-config
                key: postgres_password
          - name: POSTGRES_HOST
            valueFrom:
              configMapKeyRef:
                name: hostname-config
                key: postgres_host
          - name: TLS-CERTIFICATE
            valueFrom:
              secretKeyRef:
                name: f-tls
                key: Certificate.p12
          - name: TLS-PASSWORD
            valueFrom:
              secretKeyRef:
                name: f-tls
                key: password

This is how I create secret in Kubernetes:

kubectl create secret generic f-tls --from-file=Certificate.p12 --from-literal=password=changeit

When it's deployed I'm getting

State:         Waiting
  Reason:      CrashLoopBackOff
Last State:    Terminated
  Reason:      ContainerCannotRun
  Message:     oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:295: setting oom score for ready process caused \"write /proc/13895/oom_score_adj: invalid argument\""

When I remove the Secrets from the Deployment yaml it's working fine, but I could not understand what it the root cause of this issue. I'm using Google Cloud Platform Container Engine.

Alcatraz

is the TLS-CERTIFICATE versus TLS_CERTIFICATE a S.O. typo, or your descriptor also contains that typo? – Globular

Separately, server.ssl.key-store: ${TLS_CERTIFICATE} would be much, much, much better served by either file:///a/fs/path/Certificate.p12 or classpath:///Certificate.p12 rather than trying to inject a binary value into an environment variable. I don't know that it's your problem, but I know for sure it's not helping matters – Globular

Same "is it a question typo or a real typo" for server.ssl.keyAlias fundtr which is missing its K-V delimiter – Globular

I have managed to resolve the issue by putting p12 files inside the jar in the resource folder. And It works fine, but I want to put the certificate in kubernetes to make it flexible. – Alcatraz

Do you mean that classpath:///Certificate.p12, to include the certificate in the container image is a good option? – Alcatraz

It looks like your are getting a "oom_score_adj" error that happen when an environmental variable is not properly declared therefore probably the environment being passed is being treated as a null pointer and this is causing the issue. I suggest to review your variables set in the Spring Boot application.properties. I could help you with this if you could point out what tutorial or guide you are following ? – Erymanthus

Im following the main tutorial from https://kubernetes.io/docs/concepts/configuration/secret/. I saw another tutorial today http://software.danielwatrous.com/generate-tls-secret-for-kubernetes/ Maybe the issue is that I didnt specify the type correctly. – Alcatraz

Do you mean that classpath:///Certificate.p12, to include the certificate in the container image is a good option that's why I mentioned file:// also, but don't forget that the classpath can include folders as well as jars, so one need not bundle the p12 just to make it available on the classpath. A silly, non-production, example might be to volume mount webapps/ROOT/WEB-INF/classes/Certificate.p12 which would make it appear on the classpath without actually living in your deployment (emphasis: just a silly example of the idea) – Globular

@PhilipPetrov are you sure you are accessing correctly to the system variable in your spring code ? for instance why for "spring.datasource.username" you use "=" and for "server.ssl.key-store" you use ":" ? – Erymanthus

@Erymanthus sorry for the late response there is no difference between ":" and "=" in properties file. – Alcatraz

@MatthewLDaniel For now I follow your advice to add the p12 in the JAR, but it is ony for development state. I want to make a more stable solution so I can use it in production. Do you think using a Volumes will be good option to store P12 file? – Alcatraz

Answer 1

This answer is specific to Springboot application and that is what asked in question.

Step 1: Create a generic secret from your keystore or p12 file

kubectl create secret generic f-tls-secret --from-file=Certificate.p12 --from-literal=password=changeit

Step 2: Mount the secret to your pod using deployment object

spec:
  containers:
  - image: eu.gcr.io/..../...
    volumeMounts:
      - name: tls
        mountPath: /workspace/resources/

  volumes:
    - name: tls
      secret: 
        secretName: f-tls-secret

3. Configure SSL in application.properties file

#Setup SSL
 server.port: 8443
 server.ssl.key-store: classpath:resources/Certificate.p12
 server.ssl.key-store-password: ${TLS_PASSWORD}
 server.ssl.keyStoreType: PKCS12
 server.ssl.keyAlias fundtr

Medorra

Answer 2

This is my deployment.yaml, which uses p12 key and password stored in Kubernetes secrets, created just like in your example. Works OK for me to make SSL curl calls. I fetch the content of p12 key and password files mounted as READ ONLY volume. Hope it helps.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: deployment-name
spec:
  replicas: 3
  selector:
    matchLabels:
      app: app-name

  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  minReadySeconds: 30

  template:
    metadata:
      labels:
        app: app-name
    spec:
      volumes:
        - name: application
          emptyDir: {}
        - name: secrets
          secret:
            secretName: key.p12

      containers:
        - name:  php-fpm
          image: index.docker.io/docker_account/docker_image:image_tag
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9000

          volumeMounts:
            - name:      application
              mountPath: /app
            - name:      secrets
              mountPath: /api-p12-keys
              readOnly:  true

      imagePullSecrets:
        - name: docker-auth

Weird