Login/Register
Provide SSL certificate for dockerd for own docker registry
Asked Answered
dockersslgitlab-cidocker-registrydocker-in-docker
L

2

19

how to provide own CA Root certificate and SSL Client certificate (cert + key) to dockerd in gitlab-ci pipeline for own docker registry?

I have virtual machine (CentOS 7) and installed docker and gitlab-runner. The runner is registred as docker:dind. The setup works OK but i am having trouble connecting to my own docker registry that has certificate from my own CA, but is also using client SSL certificate. When i pass to dockerd the argument --insecure-registry=gitlab.mazel.tov:4567 it doesnt verify the CA but i still dont know how to provide the Client SSL certificate and pipile will fail with this error.

$ docker login -u gitlab-ci-token -p $CI_JOB_TOKEN https://gitlab.mazel.tov:4567
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://gitlab.mazel.tov:4567/v2/: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>400 No required SSL certificate was sent</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>No required SSL certificate was sent</center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"

I also have folder inside docker:dind container with my certs, but this approach doesnt work with dockerd? But on my computer or on server it works ok.

$ ls -la /etc/docker/certs.d/gitlab.mazel.tov\:4567/
    ca.crt
    client.cert
    client.key

I also explored dockerd --help but the option about certificates are only for the docker socket and i SSL configs for docker registry.

/etc/gitlab-runner/config.toml

[[runners]]
  name = "My Docker Runner"
  url = "https://gitlab.mazel.tov"
  token = "ac17ab5cfff675fddd059d40a3"
  tls-ca-file = "/etc/ssl/certs/myCA.pem"
  tls-cert-file = "/etc/gitlab-runner/certs/mazel.tov.pem"
  tls-key-file = "/etc/gitlab-runner/certs/mazel.tov.key"
  executor = "docker"
  [runners.docker]
    image = "docker:dind"
    privileged = true
    disable_cache = false
    volumes = ["/cache", "/etc/docker/certs.d:/etc/docker/certs.d"]
    shm_size = 0
  [runners.cache]

.gitlab-ci.yml

services:
  - name: docker:dind
    command: ["--insecure-registry=gitlab.mazel.tov:4567"]

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_HOST: tcp://docker:2375/

stages:
  - build

before_script:
  - ls -la /etc/docker/certs.d/gitlab.mazel.tov\:4567/
  - docker info
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN https://gitlab.mazel.tov:4567

build-web:
  stage: build
  script:
    - docker build ...
    - docker push ...

I dont like the --insecure-registry option neither. And my gitlab server has the client certificate configured on the Nginx level.

/etc/gitlab/gitlab.rb

nginx['ssl_client_certificate'] = "/etc/gitlab/ssl/myCA.crt"
nginx['ssl_verify_client'] = "on"

The certificates works OK on my machine and other server, i am just having trouble implementing it with gitlab-ci pipeline...

When I test the docker:dind on my mac and mount certificates it works OK, so the problem is only in the gitlab-ci pipeline? Maybe the dockerd is loaded before the mount folder occurs? 

docker run -it --rm --privileged -v ~/.docker/certs.d/:/etc/docker/certs.d/ docker:dind sh
dockerd &
docker login https://gitlab.mazel.tov:4567
** it works **
Lathi answered 24/8, 2018 at 8:9 Comment(0)
A
0

DinD

I have not tested this with client certs (only with the CA) but you are free to modify the DinD service in Gitlab, overriding its command. Therefore, you are able to inject your CA, client cert and key files before launching the actual dockerd, as indicated in the dockerd documentation on this topic.

For example:

build-something:
  image: docker
  variables:
    DOCKER_HOST: "tcp://docker:2375"
    DOCKER_TLS_CERTDIR: ""

  services:
    - name: docker:dind
      command:
        - /bin/sh
        - -c
        - >-
          mkdir -p /etc/docker/certs.d/your.registry:port &&
          echo "$CUSTOM_CA_CERT" > /etc/docker/certs.d/your.registry:port/ca.crt &&
          echo "$CLIENT_CERT" > /etc/docker/certs.d/your.registry:port/client.cert &&
          echo "$CLIENT_CERT_KEY" > /etc/docker/certs.d/your.registry:port/client.key &&
          dockerd-entrypoint.sh # additional dockerd options such as --dns 1.2.3.4 could be passed here

  script:
    # Build the image, etc
    - docker build
    - docker push ...

As an alternative... enter Buildkit

What I would do instead, is to use buildkit directly, without the docker client and daemon in between. Docker uses moby/buildkit under the hood when you launch a build, but it does not expose all the options that buildkitd and buildctl have to offer.

Using moby/buildkit directly, you have full control over the buildctl, no need for the "sidecar" Gitlab service running a dockerd daemon also means faster build start time.

I'd normally define a template job such as:

.buildkit-job:
  image:
    name: moby/buildkit
    entrypoint: [ "" ]
  variables:
    BUILDCTL_CONNECT_RETRIES_MAX: 3
    BUILDKITD_TOML: |
      # Use our own docker registry mirror to ease pulls on DockerHub
      [registry."docker.io"]
      mirrors = ["your.docker.proxy.registry.here"]

      # When using  own registries signed with the custom CAs...
      [registry."your.perhaps.broken.own.registry.here"]
      insecure = true # this is one way

      # Registry that also requires client certs
      [registry."your.also.broken.own.registry.here"]
      insecure = true
      ca=["/etc/something/the-ca.pem"]
      [[registry."your.also.broken.own.registry.here".keypair]]
        # Your TLS client cert and key
        cert="/etc/something/cert.pem"
        key="/etc/something/key.pem"
       
      # Pass your own DNS to BuildKit if you have an internal services that
      # need this during build time (internal package registries?)
      [dns]
      nameservers=["10.1.2.3"]

  before_script:
      # Inject the BuildKit configuration
      - mkdir /etc/buildkit
      - echo "$BUILDKITD_TOML" > /etc/buildkit/buildkitd.toml
      - ... also inject /etc/something/cert.pem, etc. here.

... of course there is no need to use a YAML literal for the .toml, you can use variables, inject files, etc. I used a variable here so the example is more self contained.

This method gives your full control over buildkit using the TOML file.

A simple build job would be extended like:

build-something:
  extends: .buildkit-job
  stage: build
  variables:
    REGISTRY: your.also.broken.own.registry.here
    IMAGE: super-duper-container
    PUSH: $REGISTRY/$IMAGE:latest,$REGISTRY/$IMAGE:$CI_COMMIT_SHORT_SHA
  script:
    # Build the image and push the results to our registry
    - buildctl-daemonless.sh build
      --frontend dockerfile.v0
      --local context=$PWD
      --local dockerfile=$PWD
      --export-cache type=registry,ref=$REGISTRY/$IMAGE-build-cache,mode=max
      --import-cache type=registry,ref=$REGISTRY/$IMAGE-build-cache
      --output type=image,\"name=$PUSH\",push=true

As you can see, buildctl also pushes (and knows how to push multiple tags efficiently). This example also uses the registry as backend for the build cache with mode=max, out of the scope of your question but just to show how some options are named differently (--export-cache ≃ --cache-to in buildx).

A con of this is that the buildctl command (CLI reference) is less known, while most folks are familiar with the good old docker build, keep that in mind.

As a final note, I would like to say that this is only reasonable in a setup where you definitely can't change your registry and its CA, for example in a corporate environment where you don't want to pick this battle. If in the other hand, you are in control of the registry, ditch that CA, client certs, and use Let's Encrypt or similar. You don't need to expose the registry to the internet for that, and you will save yourself and the ones after you a lot of headaches fighting obscure configuration flags.

Assurbanipal answered 10/1 at 0:21 Comment(0)
C
-1

Try this config under [[runners]]

 environment = ["DOCKER_TLS_CERTDIR="]
Chaparral answered 14/1, 2021 at 15:32 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.