Docker volume cleanup of gitlab-runner

Question

I am new to docker and jenkins. However ultimately I am creating a job in jenkins so that I can delete the volume cache of gitlab-runner which is stored in our linux machines (CentOS7)

To achieve this, I am creating a periodic job every 6 hours with the following command in jenkins:

docker volume prune -f 

However it doesn't clean up the space at all. This is the output of jenkins job :

Started by timer
Running as SYSTEM
Building remotely on buildbng17 (gitlab) in workspace /mnt/data0/jenkins/workspace/gitlab-cleanup
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] Done
[gitlab-cleanup] $ /bin/sh -xe /tmp/jenkins3799570002967825583.sh
+ find /mnt/data0/gitlab/data/backups/ -name '*.tgz' -mtime +30
/mnt/data0/gitlab/data/backups/etc-gitlab-1611415968.tgz
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Started calculate disk usage of workspace
Finished Calculation of disk usage of workspace in 0 seconds
Finished: SUCCESS

When I went to my buildbng17 machine and checked if there is any volume their, so there was volume which I had to clean up by performing

docker volume prune

The only drawback is that I need to do it manually and give that "y" when it asks for confirmation. And the data gets clean.

What command should I give in jenkins so that it will automatically clean up the volume without asking for confirmation? (because docker volume prune -f is not working)

I even tried to run the docker volume prune -f command manually in my linux machine, it still doesn't clean up the volume and shows 0kb freed up (However there is the volume which I checked from docker volume ls)

Is there any better approach we can do to automate it inside gitlab-runner itself ? Or some better feature of jenkins or docker of which I am not aware of?

The gitlab-runner keeps on running and covers the diskspace.

One more thing : We are using Gitlab and Gitlab-runner on docker.

Thanks in advance, Sameer

Embitter

Answer 1

Run these commands to do cleanup:

# Remove exited containers
docker ps -a -q -f status=exited | xargs --no-run-if-empty docker rm -v

# Remove dangling images
docker images -f "dangling=true" -q | xargs --no-run-if-empty docker rmi

# Remove unused images
docker images | awk '/ago/  { print $3}' | xargs --no-run-if-empty docker rmi

# Remove dangling volumes
docker volume ls -qf dangling=true | xargs --no-run-if-empty docker volume rm

Roadwork

Thanks a lot @Glen these commands perfectly worked for me. However can you tell me one thing what exactly does the { print $3 } means? – Embitter

awk is a text processor like 'sed'. /ago/ is a regular expression that selects the lines containing the text 'ago', this removes the header row from the docker images output. The { print $3 } is a script that prints the 3rd token found on each line, which is the Image ID value. If you changed $3 to $2 it would output the image tag and $1 would print the image repository + name. – Roadwork

@SameerAtharkar if this worked for you, would you like to mark this as an answer? Thanks – Tinner

This worked for us, too. Thanks a lot. It should be the accepted answer. – Tamas

Isn't it the same as running a single docker system prune command? – Kranz

Can someone explain why the last command sometimes removes volumes docker volume prune refuse to remove? (ie.: dangling volumes are sometimes not removed by docker volume prune) – Coherence

For docker 23: github.com/docker/cli/issues/4028#issuecomment-1429538124 – Coherence

Answer 2

GitLab Runner provides a clear-docker-cache script. It is recommended to run it weekly via cron.

Create /etc/cron.weekly/clear-docker-cache with the following contents:

#!/bin/sh
/usr/share/gitlab-runner/clear-docker-cache

Running /usr/share/gitlab-runner/clear-docker-cache will default to running clear-docker-cache prune-volumes, which basically runs docker system prune --volumes -af.

If you don't want to prune volumes, use /usr/share/gitlab-runner/clear-docker-cache prune.

If you want to add a filter, add the --filter parameter, like --filter "until=24h" to prune prior to the last 24 hours.

If you just want to see what space could be reclaimed, run /usr/share/gitlab-runner/clear-docker-cache space.

---

Another option is to use a pre-build script. GitLab CI/CD runner clean-up with pre-build scripts. It's possible to use both methods.

Tiros

Answer 3

docker system prune command clean can affect the network too

https://docs.docker.com/engine/reference/commandline/system_prune/

Bambino

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Joub