Does gitlab-runner or Docker cache the /builds Directory by Default?

Question

I am using gitlab-runner with Docker on my MacBook Pro. I'm using gitlab.com to host my git repositories. I have configured one repository to execute a pipeline whenever a commit has been pushed to the repo. The pipeline has two jobs: display some information, and clone a separate repository as a dependency.

Here is the output from the first job:

Running with gitlab-runner 12.2.0 (a987417a)
  on old_gen_runner 6fsjs96e
Using Docker executor with image gcc-arm_4_7-2013q3:3.0 ...
Using docker image sha256:5d4741a428654beb44a3c004822e4d2ceededc88f22ec1ec7f45dedf707a0302 for gcc-arm_4_7-2013q3:3.0 ...
Running on runner-6fsjs96e-project-13664495-concurrent-0 via my_laptop.local...
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/staging-fw/my-project/.git/
Created fresh repository.
From https://gitlab.com/staging-fw/my-project
 * [new branch]      master     -> origin/master
Checking out 722c2c38 as master...

Skipping Git submodules setup
$ python /scripts/info.py
Printing job info...

Builds dir: /builds

Commit Message: 
Modified yml file.

Branch: master
Project dir: /builds/staging-fw/my-project
$ ls /builds
my-project

Here is the output from the second job:

Running with gitlab-runner 12.2.0 (a987417a)
  on old_gen_runner 6fsjs96e
Using Docker executor with image gcc-arm_4_7-2013q3:3.0 ...
Using docker image sha256:5d4741a428654beb44a3c004822e4d2ceededc88f22ec1ec7f45dedf707a0302 for gcc-arm_4_7-2013q3:3.0 ...
Running on runner-6fsjs96e-project-13664495-concurrent-0 via my_laptop.local...
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/staging-fw/my-project/.git/
Checking out 722c2c38 as master...

Skipping Git submodules setup
$ git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/staging-fw/my-dependencies.git /builds/dependencies
Cloning into '/builds/dependencies'...
Job succeeded

As you can see, the first time the pipeline is created, it is successful.

If I run it again by pushing new changes, here is the output from the first job:

Running with gitlab-runner 12.2.0 (a987417a)
  on old_gen_runner 6fsjs96e
Using Docker executor with image gcc-arm_4_7-2013q3:3.0 ...
Using docker image sha256:5d4741a428654beb44a3c004822e4d2ceededc88f22ec1ec7f45dedf707a0302 for gcc-arm_4_7-2013q3:3.0 ...
Running on runner-6fsjs96e-project-13664495-concurrent-0 via my_laptop.local...
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/staging-fw/my-project/.git/
From https://gitlab.com/staging-fw/my-project
   722c2c3..f321b75  master     -> origin/master
Checking out f321b75f as master...

Skipping Git submodules setup
$ python /scripts/info.py
Printing job info...

Builds dir: /builds

Commit Message: 
Modified yml file.

Branch: master
Project dir: /builds/staging-fw/my-project
$ ls /builds
my-project
dependencies
Job succeeded

As you can see, the dependencies folder is still there? Not only that, but instead of saying Initialized empty Git repository as it did in the first run of the first job, it now says Reinitialized existing Git repository.

Here is what happens on the second job during the second run:

Running with gitlab-runner 12.2.0 (a987417a)
  on old_gen_runner 6fsjs96e
Using Docker executor with image gcc-arm_4_7-2013q3:3.0 ...
Using docker image sha256:5d4741a428654beb44a3c004822e4d2ceededc88f22ec1ec7f45dedf707a0302 for gcc-arm_4_7-2013q3:3.0 ...
Running on runner-6fsjs96e-project-13664495-concurrent-0 via my_laptop.local...
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/staging-fw/my-project/.git/
Checking out f321b75f as master...

Skipping Git submodules setup
$ git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/staging-fw/my-dependencies.git /builds/dependencies
fatal: destination path '/builds/dependencies' already exists and is not an empty directory.
ERROR: Job failed: exit code 1

My questions are:

1. Is the /builds directory created for each individual container in Docker? This was my original understanding, that each container within Docker would have its own separate /builds directory.

2. Is the /builds directory shared for all Docker containers? I cannot find any information on this.

2a. If the /builds directory is shared for all Docker containers, then where is it and who created it?

Renunciation

Answer 1

The default Git strategy for GitLab CI/CD is 'fetch'. If the same static runner is used for the same pipeline over and over, GitLab won't do a fresh clone but will just fetch changes on subsequent runs.

If you want a fresh environment each time you want the 'clone' strategy. There are two ways you can change this behavior:

1. Change the project default to 'clone' by going to project Settings -> CI/CD -> General pipelines -> Set 'Git strategy for pipelines' to git clone
2. Change the strategy via .gitlab-ci.yml:

variables:
  GIT_STRATEGY: clone

You can learn more about this setting in the GitLab documentation - Git strategy

Anhwei

Answer 2

1. Is the /builds directory created for each individual container in Docker? This was my original understanding, that each container within Docker would have its own separate /builds directory.

Short answer: when using the docker executor, it depends on GIT_STRATEGY.

/builds is a docker volume, which is mounted from the host.

This is somehow mentioned in the doc:

• https://docs.gitlab.com/runner/executors/docker.html#the-builds-and-cache-storage
• https://docs.gitlab.com/runner/executors/docker.html#the-persistent-storage

Whether the volume persists or not depends on the GIT_STRATEGY of the job:

• if GIT_STRATEGY=="fetch" the volume persists and is shared with successive single-use containers which run jobs (on this host, for this project).
• otherwise, the volume is considered "temporary" and removed at the end of the job.

This can be seen in gitlab-runner's implementation:

func (e *executor) createBuildVolume() error {
    ...
    if e.Build.GetGitStrategy() == common.GitFetch {
        err = e.volumesManager.Create(e.Context, jobsDir)
        ...
    } else {
        err = e.volumesManager.CreateTemporary(e.Context, jobsDir)
    }
...
}

When your runner is building something, you can also inspect its container and find out which volumes are used (here on linux with gitlab-runner 14.10, running a job with GIT_STRATEGY=="fetch") :

# docker container ls
9c981d9ec900   7d926f4bcaec   "sh -c 'if [ -x /usr…"   8 minutes ago   Up 8 minutes             runner-dfciu5nh-project-3374-concurrent-0-b73d69d36b7291e4-build-2

# docker container inspect 9c981d9ec900 | jq .[0].Mounts
[
  {
    "Type": "volume",
    "Name": "runner-dfciu5nh-project-3374-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70",
    "Source": "/var/lib/docker/volumes/runner-dfciu5nh-project-3374-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70/_data",
    "Destination": "/cache",
    "Driver": "local",
    "Mode": "z",
    "RW": true,
    "Propagation": ""
  },
  {
    "Type": "volume",
    "Name": "runner-dfciu5nh-project-3374-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8",
    "Source": "/var/lib/docker/volumes/runner-dfciu5nh-project-3374-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8/_data",
    "Destination": "/builds",
    "Driver": "local",
    "Mode": "z",
    "RW": true,
    "Propagation": ""
  }
]

Hence, the docker volume named runner-dfciu5nh-project-3374-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8 is mounted from /var/lib/docker/volumes/runner-dfciu5nh-project-3374-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8/_data on the host to /builds in the container.

2. Is the /builds directory shared for all Docker containers? I cannot find any information on this.

If GIT_STRATEGY=="fetch" the /builds volume does persist and is reused for subsequent builds of the same project. Note how the project id (3374 in my example) is in the volume name. If you runner accepts several concurrent jobs, you runner might also have several /builds volumes per project (hence the concurrent id in the volume name).

If GIT_STRATEGY!="fetch", the volume does not persist and is not shared with any container.

2a. If the /builds directory is shared for all Docker containers, then where is it and who created it?

It is created by gitlab-runner itself, lazily (ie. if needed when running a given job).

By the way, in order to clear the build directory you can:

• use a GIT_STRATEGY different from "fetch".
• use GIT_STRATEGY=="fetch" and let gitlab-runner do it before running your job, when it checkouts the source. This is configurable with GIT_CLEAN_FLAGS. This can also be done at the end of the job with the FF_ENABLE_JOB_CLEANUP feature flag.
• configure your runner with the disable_cache docker-specific runner setting
• do rm -rf at the beginning and/or end of the job (in the .gitlab-ci.yml file)
• remove the docker volumes on the host. Either using or taking inspiration from the clearing-docker-cache script. This is more for garbage collection than for ensuring a fresh build environment though.

I also suspect you could tweak the builds_dir and volumes runner settings to use a tmpfs

(I haven't tried it).

Turgor

Your answer is ten times better than the official Gitlab documentation. – Berner

Notice that GIT_STRATEGY none is using a temporary volume. One needs to put some [runners.docker] volumes = ["/builds"] to prevent the runner from destroying the directory. This is an undocumented feature/behavior you can see in the same function you showed. – Berner

Answer 3

Go to your Project menu : CI/CD -> Pipeline And click to [Clear runner caches] It's works for me !

Kean