Docker Registry 2.0 - how to delete unused images?

Question

We updated our private docker registry to the official Registry 2.0. This version can now delete docker images identified by a hashtag (see https://docs.docker.com/registry/spec/api/#deleting-an-image) but I still don't see a way to cleanup old images.

As our CI server is continously producing new images, I would need a method to delete all images from the private registry which are no longer identified by a named tag.

If there's no built-in way to achieve this, I think a custom script could possibly work, but I don't see a v2 API method either to list all stored hashtags of an image..

How can I keep my private registry clean? Any hints?

Romanticism

Actually, the situation is worse than you think. The spec is not yet implemented; calling DELETE returns a 400. See github.com/docker/distribution/issues/422 – Musket

private registry has a long way to go – Dismissal

Hello from the future (4 years later) - is there a good method to deal with this nowadays? The options seem abut the same as they did when this question was active. – Truly

Answer 1

Deletion of images (you can keep 10 last versions, like I do in my CI) is done in three steps:

1. Enable image deletion by setting environment variable REGISTRY_STORAGE_DELETE_ENABLED: "true" and passing it to docker-registry

2. Run below script (it will delete all images and tags but keep last 10 versions)

   registry.py -l user:pass -r https://example.com:5000 --delete --num 10

3. Run garbage collection (you can put it into your daily cron task)

   docker-compose -f [path_to_your_docker_compose_file] run registry bin/registry garbage-collect /etc/docker/registry/config.yml

registry.py can be downloaded from the link below, it also allows listing images, tags and layers, as well as deleting a particular image and/or tag.

https://github.com/andrey-pohilko/registry-cli

Before garbage collection my registry folder was 7 Gb, after I ran the above steps it deflated down to 1 Gb.

Silsbye

The thing with this solution is that it can't deal with... untagged revisions (?). Those you push to a registry without specifying a tag. @Silsbye Can you confirm? I was able to free 120 Gb by deleting those. But there were issues, I'm about to add a comment to that answer. – Cheremkhovo

@Cheremkhovo probably you're right. My script is not intended to clean up all revisions, like in the solution you used. Known issue with the script is that it cannot delete the entire image from registry. But, as you can see, the original question is about how to clean old tags from CI server, not all the revisions or all repos. So, both solutions are good, they just serve different purposes. – Silsbye

I don't want to delete all revisions (that would basically delete a tag), not to say all repositories. It's your script that basically deletes all revisions of tags by deleting tags themselves. And I'm in no way trying to devalue your answer. I'm just adding details regarding my case. From what I can see the registry.py uses the api (/v2/_catalog for repositories, – Cheremkhovo

.../v2/REPO/tags/list for tags). But the things is, every tag generally has more than one revision. One can see those at /var/lib/docker/volumes/registry/_data/docker/registry/v2/repositories/REPO/_manifests/tags/TAG/index. And those can be deleted using DELETE /v2/REPO/manifests/sha256:HASH. – Cheremkhovo

...But the filesystem doesn't seem to accurately reflect the state of affairs, and that might get you by deleting tags, not just old revisions of a tag (you delete all revisions but the last, but the last one exists only on disk). At least, that's the best explanation of what went wrong that I've got so far. Hopefully, the official solution does what I think it does. – Cheremkhovo

Answer 2

Regarding your question:

I would need a method to delete all images from the private registry which are no longer identified by a named tag

A new version of the docker registry in distribution/registry:master has this nice feature! However, you won't be able to trigger it from the API.

Anyway, you will be able to clean all untagged manifests in your registry, meaning that every overwritten tag won't leave old manifests and blobs in the registry. Every "unused" layer will be cleaned by the Registry Garbage Collectior.

You just have to run a docker exec:

docker exec ${container_id} registry garbage-collect \ 
  /path/to/your/registry/config.yml \
  --delete-untagged=true

Looking at this garbage-collect binary help:

Usage: 
  registry garbage-collect <config> [flags]
Flags:
  -m, --delete-untagged=false: delete manifests that are not currently referenced via tag
  -d, --dry-run=false: do everything except remove the blobs
  -h, --help=false: help for garbage-collect

You can have a look at the github PR. It's been merged and usable with distribution/registry, tag master as of 2018-02-23. It supersedes the docker/docker-registry project with a new API design, focused around security and performance...

I did use this feature today and recovered 89% of registry space (5.7 GB vs. 55 GB). Then I switched back to stable registry.

Moffatt

The changes from the PR are supposed to be in the official image now. – Cheremkhovo

Right now this should be the accepted answer. It does exactly what was required. If someone is running a private repo, the only thing required is to just pull the new image docker pull registry:2, stop the container docker-compose stop registry, remove it docker-compose rm registry and recreate it docker-compose up -d registry. The above will work if you use docker-compose and execute in the directory with docker-compose.yml AND your container is names registry :) – Jejunum

The garbage collector is now in Docker Registry v2.7+, so any freshly installed Docker Registry should have it. – Alto

This is a good answer. However I had no idea where or what my "registry file" was. I found its location by doing a docker exec -it <docker-registry-container-name> /bin/sh Then performed a find / -iname "*.yml" Which uncovered the location of the file. – Telemechanics

Can this be done automatically somehow? – Adaminah

Looks like the config file is at /etc/docker/registry/config.yml by default – Shiv

Answer 3

I host regestry in docker container with name docker-registry_registry_1 from image: registry:2

I just run garbage-collect with -m

docker exec docker-registry_registry_1 bin/registry garbage-collect /etc/docker/registry/config.yml -m

Terse

Answer 4

This is doable, although ugly. You need to be running (I think) registry 2.3 or greater, and have enabled deleting (REGISTRY_STORAGE_DELETE_ENABLED=True env var or equivalent). The example commands below assume a local filestore in /srv/docker-registry, but I'd be surprised if something equivalent couldn't be cooked up for other storage backends.

For each repository you wish to tidy up, you need to enumerate the digest references that are no longer required. The easiest way to do this is per-tag, using latest as an example in this case, you'd do something like:

ls -1tr /srv/docker-registry/v2/repositories/<repo>/_manifests/tags/latest/index/sha256 \
| tail -n +3

This will list all but the three most recent digests pushed to the latest tag. Alternately, if you don't care about tags so much, but just want to keep the last few references pushed, you can do:

ls -1t /srv/docker-registry/v2/repositories/<repo>/_manifests/revisions/sha256 \
| tail -n +3

Then, you just delete the references you don't need:

for hash in $(ls -1t /srv/docker-registry/v2/repositories/<repo>/_manifests/tags/latest/index/sha256 | tail -n +3); do
  curl -X DELETE https://registry:5000/v2/<repo>/manifests/sha256:$hash
done

Finally, you need to do a GC run, because the registry implements "soft deletes", which doesn't actually delete anything, it just makes it unavailable:

docker exec docker-registry /bin/registry \
  garbage-collect /path/to/config.yml

Yes, this is all messy as hell, grovelling around in the backend storage, because there's no API method to enumerate all digests associated with a given tag, but that's the way the cookie crumbles.

Typify

Minor clarification: the flag is REGISTRY_STORAGE_DELETE_ENABLED, not ..._STORE_.... See docs.docker.com/registry/configuration/#/… for more details. – Illconditioned

In v2.6 the directory structure is .../_manifest/tags/<tag>/index/.... – Passmore

@Typify It somehow deleted a tag (latest) of one of my images. Also, state of the filesystem doesn't seem to reflect the state of affairs. After deleting unneeded manifests, and running garbage collector, I can still see at least some manifests that are supposed to be gone at /srv/docker-registry/v2/repositories/<repo>/_manifests/tags/latest/index/sha256. – Cheremkhovo

It doesn't surprise me that things have changed in the time since I wrote that answer. I no longer use the Docker registry for anything I don't absolutely have to. – Typify

I'm not sure they have changed. I might just be lucky enough to run into the case where it doesn't work properly. My conjecture is that filesystem doesn't fully reflect state of affairs. And when you delete all but the last revision, the latter might be only on disk. So you've just deleted all the revisions, and the tag disappears as well. Hopefully, the official solution does what I think it does. What do you use by the way? – Cheremkhovo

Answer 5

There is some discussion happening to design this - right now, there is no layer cleanup tool / endpoint.

I would encourage you to go to:

• https://github.com/docker/distribution/issues/106
• https://github.com/docker/distribution/issues/210

and/or reach out on Freenode IRC on #docker-distribution for more.

Penology

so assuming that we burn over 50Gb disk space a day in our CI server, what's your suggested solution to the problem? That IRC channel also looks pretty quiet to me :) – Romanticism

Short terms, here are suggestions: if that's "throwable" content, you can wipe out your entire storage once in a while. If that's not acceptable, 50GB a day makes for 5T in three months - maybe some cheap storage solution can work out? Also, I would expect a real solution coming in the next couple of months. Sorry for not having a better solution right away... – Penology

@MangledDeutz any news on this? it's been several months now.. thanks! – Kessiah

@Kessiah There is now a pending PR to officially support GC here: github.com/docker/distribution/pull/1386 - and there are community maintained solutions as well. Cheers. – Penology

@MangledDeutz great news! thanks, I hope it can be merged back soon – Kessiah

GC is fine and dandy, except that nothing actually deletes the references that would allow a GC run to nuke anything. – Typify

I suggest you reach out to forums or irc for help on these. – Penology

Answer 6

I pieced together various parts of this thread and created an easy to use cleanup script in bash You can check it out in this gist cleanup.sh

Womanhood

Thanks for your script. This worked for me with some modifications: 1) The for hash in loop assumes that repo names are a single string, while mine are named project/repo, so I had to add another inner loop to navigate the extra folder level. 2) Added authentication to the curl command with -u usr:pwd. 3) Added REGISTRY_STORAGE_DELETE_ENABLED=true to the env vars for running the registry. – Cost

@Cost It's better to use -n, --netrc option in place of -u. – Cheremkhovo

Answer 7

I was looking for the same functionality in the registry v2 api but only found soft deleting which is not what I was looking for. While researching I found the Github project delete-docker-registry-image which removes the actual files from the mounted volume via a bash script. Not tested it maybe useful...

Wessels

it seems to be doing the trick (in my environment :) I had to create a symlink 'ln -s /var/lib/docker/volumes/c49189f29d8bd93f644438dee774685790687a67c576eb1349cbfe218e14fc20/_data /opt/registry_data' as I run my registry inside a container and the cryptic string is the mount of the container (to be retrieved by the command docker inspect --format '{{range .Mounts}}{{.Name}}{{end}}' <containername>) – Stationery

Answer 8

For removing unsed images, three steps manually on these sequence:

1. docker rmi -f **imageid**

2. rm -Rf /home/**homedirectory**/docker-registry/data/docker/registry/v2/repositories/**yoursystemname**/**yourimagename**/_manifests/tags/**image version**/

3. docker exec $(docker ps -q) bin/registry garbage-collect /etc/docker/registry/config.yml -m

*Pay attention:

** You must execute those commands (above) in test environment, because if you commit any mistake or didn't understand any step, you don't damage your production environment.

** You can schedule those commands (above) using crontab as root. In the step 3) you must execute removing "-it", as result: docker exec $(docker ps -q) bin/registry garbage-collect /etc/docker/registry/config.yml -m`.

It works for me for more than 6 months.

Geostatic

Answer 9

Kubernetes equivalent:

kubectl exec --tty --stdin registry-0 -n devops -- registry garbage-collect /etc/docker/registry/config.yml --delete-untagged=true

Casabonne

that would be quite particular to your setup – Shiv