How to customise config.toml on Kubernetes?

Question

I'm have a Gitlab cloud connected to a k8s cluster running on Google (GKE). The cluster was created via Gitlab cloud.

I want to customise the config.toml because I want to fix the cache on k8s as suggested in this issue.

I found the config.toml configuration in the runner-gitlab-runner ConfigMap. I updated the ConfigMap to contain this config.toml setup:

  config.toml: |
    concurrent = 4
    check_interval = 3
    log_level = "info"
    listen_address = '[::]:9252'
    [[runners]]
      executor = "kubernetes"
      cache_dir = "/tmp/gitlab/cache"
      [runners.kubernetes]
        memory_limit = "1Gi"
        [runners.kubernetes.node_selector]
          gitlab = "true"
        [[runners.kubernetes.volumes.host_path]]
          name = "gitlab-cache"
          mount_path = "/tmp/gitlab/cache"
          host_path = "/home/core/data/gitlab-runner/data"

To apply the changes I deleted the runner-gitlab-runner-xxxx-xxx pod so a new one gets created with the updated config.toml.

However, when I look into the new pod, the /home/gitlab-runner/.gitlab-runner/config.toml now contains 2 [[runners]] sections:

listen_address = "[::]:9252"
concurrent = 4
check_interval = 3
log_level = "info"

[session_server]
  session_timeout = 1800

[[runners]]
  name = ""
  url = ""
  token = ""
  executor = "kubernetes"
  cache_dir = "/tmp/gitlab/cache"
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = ""
    namespace = ""
    namespace_overwrite_allowed = ""
    privileged = false
    memory_limit = "1Gi"
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.node_selector]
      gitlab = "true"
    [runners.kubernetes.volumes]

      [[runners.kubernetes.volumes.host_path]]
        name = "gitlab-cache"
        mount_path = "/tmp/gitlab/cache"
        host_path = "/home/core/data/gitlab-runner/data"

[[runners]]
  name = "runner-gitlab-runner-xxx-xxx"
  url = "https://gitlab.com/"
  token = "<my-token>"
  executor = "kubernetes"
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = "ubuntu:16.04"
    namespace = "gitlab-managed-apps"
    namespace_overwrite_allowed = ""
    privileged = true
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.volumes]

The file /scripts/config.toml is the configuration as I created it in the ConfigMap. So I suspect the /home/gitlab-runner/.gitlab-runner/config.toml is somehow updated when registering the Gitlab-Runner with the Gitlab cloud.

If if changing the config.toml via the ConfigMap does not work, how should I then change the configuration? I cannot find anything about this in Gitlab or Gitlab documentation.

Counteraccusation

Answer 1

Inside the mapping you can try to append the volume and the extra configuration parameters:

# Add docker volumes
cat >> /home/gitlab-runner/.gitlab-runner/config.toml << EOF

      [[runners.kubernetes.volumes.host_path]]
        name = "var-run-docker-sock"
        mount_path = "/var/run/docker.sock"
EOF

I did the runner deployment using a helm chart; I guess you did the same, in the following link you will find more information about the approach I mention: https://gitlab.com/gitlab-org/gitlab-runner/issues/2578

If after appending the config your pod is not able to start, check the logs, I did test the appending approach and had some errors like "Directory not Found," and it was because I was appending in the wrong path, but after fixing those issues, the runner works fine.

Boccherini

Great idea. It worked for me like a charm. Thanks a lot – Hackery

How do you change the config of a runner that is already running? – Lactoscope

Answer 2

Seems to me you should be modifying config.template.toml (within your relevant configmap, that is)

Sopping

Answer 3

If installing gitlab-runner from the helm chart gitlab/gitlab-runner (at least for chart version 0.52.0), enter config.toml values in an additional helm values file.

The values.yaml that produces the configuration desired in the OP would contain:

gitlabUrl: https://gitlab.com
runnerToken: <mytoken>
concurrent: 4
checkInterval: 3
logLevel: "info"
runners:
  config: |
    [[runners]]
      cache_dir = "/tmp/gitlab/cache"
      [runners.kubernetes]
        memory_limit = "1Gi"
        namespace = "{{.Release.Namespace}}"
        image = "ubuntu:16.04"
      [runners.kubernetes.node_selector]
        gitlab = "true"
      [[runners.kubernetes.volumes.host_path]]
        name = "gitlab-cache"
        mount_path = "/tmp/gitlab/cache"
        host_path = "/home/core/data/gitlab-runner/data"
  executor: kubernetes

Sen

Answer 4

If you want modify existing config.toml in /home/gitlab-runner/.gitlab-runner you need to set environment variables in deployment. For example, this is default set of variables in case you have installed gitlab-runner by pressing install button in gitlab.

 Environment:
       CI_SERVER_URL:                                    http://git.example.com/
       CLONE_URL:                                        
       RUNNER_REQUEST_CONCURRENCY:                       1
       RUNNER_EXECUTOR:                                  kubernetes
       REGISTER_LOCKED:                                  true
       RUNNER_TAG_LIST:                                  
       RUNNER_OUTPUT_LIMIT:                              4096
       KUBERNETES_IMAGE:                                 ubuntu:16.04
       KUBERNETES_PRIVILEGED:                            true
       KUBERNETES_NAMESPACE:                             gitlab-managed-apps
       KUBERNETES_POLL_TIMEOUT:                          180
       KUBERNETES_CPU_LIMIT:                             
       KUBERNETES_CPU_LIMIT_OVERWRITE_MAX_ALLOWED:       
       KUBERNETES_MEMORY_LIMIT:                          
       KUBERNETES_MEMORY_LIMIT_OVERWRITE_MAX_ALLOWED:    
       KUBERNETES_CPU_REQUEST:                           
       KUBERNETES_CPU_REQUEST_OVERWRITE_MAX_ALLOWED:     
       KUBERNETES_MEMORY_REQUEST:                        
       KUBERNETES_MEMORY_REQUEST_OVERWRITE_MAX_ALLOWED:  
       KUBERNETES_SERVICE_ACCOUNT:                       
       KUBERNETES_SERVICE_CPU_LIMIT:                     
       KUBERNETES_SERVICE_MEMORY_LIMIT:                  
       KUBERNETES_SERVICE_CPU_REQUEST:                   
       KUBERNETES_SERVICE_MEMORY_REQUEST:                
       KUBERNETES_HELPER_CPU_LIMIT:                      
       KUBERNETES_HELPER_MEMORY_LIMIT:                   
       KUBERNETES_HELPER_CPU_REQUEST:                    
       KUBERNETES_HELPER_MEMORY_REQUEST:                 
       KUBERNETES_HELPER_IMAGE:

Modify existing values or add new ones - it will appear in correct section of config.toml.

Ovariotomy