How to add trusted root CA to Docker alpine

Question

Suppose I am at network where there is MITM SSL swaping firewall (google.com is not issued by Google, but reissued by custom CA root authority) some more details here https://security.stackexchange.com/questions/107542/is-it-common-practice-for-companies-to-mitm-https-traffic .

I have simple Dockerfile:

FROM alpine:latest
RUN apk --no-cache add curl

It fails badly with error with SSL errors

 => ERROR [2/2] RUN apk --no-cache add curl                                                                                                                                    1.0s
------
 > [2/2] RUN apk --no-cache add curl:
#5 0.265 fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
#5 0.647 140037857143624:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913:
#5 0.649 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/main: Permission denied
#5 0.649 fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
#5 0.938 140037857143624:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913:
#5 0.940 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/community: Permission denied
#5 0.941 ERROR: unable to select packages:
#5 0.942   curl (no such package):
#5 0.942     required by: world[curl]
------
executor failed running [/bin/sh -c apk --no-cache add curl]: exit code: 1

Every tutorial at Internet says that I can add own "trusted" root certificate and run update-ca-certificates.

But it can be added by "apt add" only. This situation seems to me as "chicken-egg" problem.

FROM alpine:latest
USER root
RUN apk --no-cache add ca-certificates \
  && update-ca-certificates

Error is similar

=> ERROR [2/2] RUN apk --no-cache add ca-certificates   && update-ca-certificates                                                                                             1.0s
------
 > [2/2] RUN apk --no-cache add ca-certificates   && update-ca-certificates:
#5 0.269 fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
#5 0.662 140490932583240:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913:
#5 0.663 fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
#5 0.663 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/main: Permission denied
#5 0.929 140490932583240:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1913:
#5 0.931 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/community: Permission denied
#5 0.932 ERROR: unable to select packages:
#5 0.933   ca-certificates (no such package):
#5 0.933     required by: world[ca-certificates]
------
executor failed running [/bin/sh -c apk --no-cache add ca-certificates   && update-ca-certificates]: exit code: 1

Is there some other solution how to install update-ca-certificates tool? Or am I missing something?

Thx

See @kthompso answer for working solution.

Working solution (with update-ca-certificates commnad) based on @kthompso answer and info from unable to add certificates to alpine linux container

FROM alpine:latest

USER root

# To be able to download `ca-certificates` with `apk add` command
COPY my-root-ca.crt /root/my-root-ca.crt
RUN cat /root/my-root-ca.crt >> /etc/ssl/certs/ca-certificates.crt

# Add again root CA with `update-ca-certificates` tool
RUN apk --no-cache add ca-certificates \
    && rm -rf /var/cache/apk/*
COPY my-root-ca.crt /usr/local/share/ca-certificates
RUN update-ca-certificates

RUN apk --no-cache add curl

Edit: One solution I have in my mind is to use curl docker image with -k option and download .apk with those certificates and tools. Install it as local file. Add my root CA certificate and run update-ca-certificates. It sounds super crazy, so I think that have to be better solution :)

Middleman

Why do you suggest adding it "again" at the end? "# Add again root CA with update-ca-certificates tool" – Mariannmarianna

First step is to be able download anythink using apk. Second step (the step you are asking) is to download ca-certificates tool and then add CA standard way with calling update-ca-certificates. First step is more or less hack. – Middleman

Yes but I read that the update-ca-certificates tool just appends certificates to that file like we did in the first step. It seems like it doesn't add any value. What else does it do? – Mariannmarianna

It is more standard way how to add certificates. First step is just only hack to have that MITM CA in system and continue to work with other commands. Commands could be same, but mangling those file by "hand" seems dangerous to me. – Middleman

but you DO mangle the file by hand.... Also, shouldn't you first remove the cert before adding it again? Now it's in their twice (or is update-ca-certificates smart?) – Earthly

It is "smart," but I agree that it's not necessary. update-ca-certificates does apparently call post update hooks (if there are any). In this case I suspect it would not pick up the self signed cert as an "add" though because it's already in the file. manpages.ubuntu.com/manpages/xenial/man8/… – Unconventional

This helped me fix the x509 unknown authority error when running a golang app in a docker container. – Negro

There is an error in the COPY command of your working solution. Taken from the Docker docs: If <dest> doesn't end with a trailing slash, it will be considered a regular file and the contents of <src> will be written at <dest>. Thus your COPY command should be COPY my-root-ca.crt /usr/local/share/ca-certificates/ instead. – Burrell

Answer 1

Append your self-signed cert to /etc/ssl/certs/ca-certificates.crt manually.

Assuming you have the self-signed certificate in a file in your build directory called my-cert.pem:

FROM alpine:latest
  
COPY my-cert.pem /usr/local/share/ca-certificates/my-cert.crt

RUN cat /usr/local/share/ca-certificates/my-cert.crt >> /etc/ssl/certs/ca-certificates.crt && \
    apk --no-cache add \
        curl

Note: When you're using update-ca-certificates, you need to place your cert file into /usr/local/share/ca-certificates/ first. Otherwise it will be removed from /etc/ssl/certs/ca-certificates.crt the first time you run update-ca-certificates.

Unconventional

I am getting same error with apk update. From my understanding it downloads current APKINDEX.tar.gz and it meets the "invalid" SSL root certificate. What you suggest is similar I have found on the Internet, but it does not solve the problem. But thx, anyways. – Middleman

Ah ok, gotcha. You should be able to just cat it into your root ca file then, updated my answer. – Unconventional

I found that with this modification you are not able to apk add ca-certificates with run of update-ca-certificates... afte this command when you run apk add curl you would get same error. So it seems to me that update-ca-certificates rollbacks that modification with cat. I have modified question with working answer based on your answer. – Middleman

Yes, as I mentioned -- if you are going to still run update-ca-certificates for some reason you need to copy your certificate into /usr/local/share/ca-certificates/ first. Otherwise, update-ca-certificates will find the presence of the certificate in your root ca file, but not in a trusted location, and assume it should be removed. That's why the COPY I provided begins by moving the self signed cert into that directory and not a random location like /root. – Unconventional

Just checking, when you said self signed certificate, is it on the container or on the host? I thought it needs to copy the local certificate from the host to the container (to the right place of course). – Orchestral

@Orchestral In the above example, the certificate file is on the host and gets copied into the image during the build process (with the COPY command). You could also pull it from somewhere else, for example a URL, using the ADD command (or curl...). Commonly you might do something like store your public cert in a read-only S3 bucket and pull it into your images during the build. – Unconventional

Answer 2

Here is the example of alpine docker images.

FROM haproxy:2.9.4-alpine
ADD cacert.crt /usr/local/share/ca-certificates/cacert.crt
USER root
RUN apk update \
&& apk upgrade --available \
&& update-ca-certificates

Decennium

update-ca-certificates should go first:) – Bhang

I don't know why but update-ca-certificates is completely not working in this version of Alpine. – Cornet

Answer 3

Another option would be to install update-ca-certificates using the --no-check-certificate added in Alpine 3.18, then run update-ca-certificates to update /etc/ssl/certs/ca-certificates.crt:

RUN apk --no-cache add --no-check-certificate ca-certificates \
    && update-ca-certificates

See: https://mcmap.net/q/244253/-how-to-disable-ssl-verification-in-alpine-39-s-apk

Millen

It's a bad idea to disable certificate checking. – Linguini