Delete Tags from a private docker registry

Question

I am a new docker user, trying to learn something about docker. I have a private docker registry. Inside it I have some images (example: I1,I2,I3,I4,I5). A lot of those images have tags inside it. Lets consider the following ones:

Image I1 - Tags: T11, T12, T13
Image I2 - Tags: T21, T22, T23, T24
Image I3 - Tags: T31, T32
Image I4 - Tags: T41, T42, T43, vT44
Image I5 - Tags: T51, T52

I want to delete, lets say, only Tags: T12, T22 and T52, from docker registry.

How can I achieve it?

I´ve been reading a lot of posts but I still haven´t figure this out. This ones seem to have some valuable content: https://docs.docker.com/registry/spec/api/#deleting-an-image Way to delete the images from Private Docker Registry Although after even reading it I still could not figure out the way to do it.

If anyone can point me in the right direction I would appreciate it.

Revell

Answer 1

This is a bit more complex than it seems on the surface if you want to delete a tag and not an image manifest. First a prerequisite, your registry needs to permit deleting images which isn't turned on by default in the registry:2 image. The easy way to enable that is setting the environment variable REGISTRY_STORAGE_DELETE_ENABLED=true on the container.

Next, realize the difference between a tag and an image manifest. The manifest is represented by a digest, and is the json data that points to the image config and layers. A tag points to a manifest, but multiple tags may point to the same manifest, and a manifest may not have any tags pointing to it. If you delete a manifest, that also removes all tags that point to the manifest, so you need to be careful when deleting tags that you don't accidentally delete a manifest referenced by tags you want to keep.

Therefore, the normal way to do this has issues. That normal way is to query the registry for the digest of the manifest you want to delete, and then delete that digest. You can get that digest from the headers:

acceptM="application/vnd.docker.distribution.manifest.v2+json"
acceptML="application/vnd.docker.distribution.manifest.list.v2+json"
curl -H "Accept: ${acceptM}" \
     -H "Accept: ${acceptML}" \
     -I -s "https://registry.example.org/v2/${repo}/manifests/${tag}" 

And then a delete request on that digest would delete the manifest, along with all tags that also point to it:

curl -H "Accept: ${acceptM}" \
     -H "Accept: ${acceptML}" \
     -X DELETE -s "https://registry.example.org/v2/${repo}/manifests/${digest}" 

However, if you want to delete just the tag, there is a delete tag API in distribution-spec, but few registries have implemented it. That delete would look like:

curl -H "Accept: ${acceptM}" \
     -H "Accept: ${acceptML}" \
     -X DELETE -s "https://registry.example.org/v2/${repo}/manifests/${tag}" 

For registries that don't support this, the best solution I've found is to push a dummy manifest that replaces the tag, and then delete that dummy manifest. That gets to be a bit much to handle with curl, there's more media type headers I haven't been including, and this doesn't talk about authentication. For all of those challenges, I turn to writing it in Go. My own tool for that is regclient, and others like skopeo and crane also exist. From regclient, the regctl command to do this looks like:

regctl tag rm registry.example.org/image1:T12
regctl tag rm registry.example.org/image2:T22
regctl tag rm registry.example.org/image5:T22

Once the image is deleted, you likely want to clean the storage that is used, and for that you need to run a garbage collection while no other pushes are in progress (some will disable the registry or wait until a time they know uploads will not run). With the registry:2 image, a GC command looks like:

docker exec registry /bin/registry garbage-collect \
  /etc/docker/registry/config.yml --delete-untagged

which will delete all untagged manifests, along with any unreferenced blobs.

Caution: untagged manifests in the distribution registry currently includes all child manifests of a multi-platform image. This means deleting untagged manifests will likely lead to data-loss if you have multi-platform images in your registry. There's issue 3178 to track when this will be resolved.

Protect

Hi, I tried the commando you provided me for "delete just the tag" but I got this error: {"errors":[{"code":"DIGEST_INVALID","message":"provided digest did not match uploaded content"}]} Any ideas? – Revell

@Fr0zt is that the regctl command or the curl? Assuming it's the curl, most likely the registry doesn't support that API and it's looking for the digest (from the above curl example). – Protect

I tried using this command: curl -H "Accept: ${acceptM}" \ -H "Accept: ${acceptML}" \ -X DELETE -s "registry.example.org/v2/${repo}/manifests/${tag}" – Revell

Any other ideas how to solve this? I mat try the other command: curl -H "Accept: ${acceptM}" \ -H "Accept: ${acceptML}" \ -X DELETE -s "registry.example.org/v2/${repo}/manifests/${digest}" Not sure it will work. In this case, assuming there is only one Docker-Content-Digest for each tag it should do what i want I guess (delete a tag for each Docker-Content-Digest). – Revell

You can delete the digest and all tags pointing to that digest would be deleted, or you can use regctl which deletes just the one tag you want to delete. – Protect

ok. So if I use the curl command to delete using digest: curl -H "Accept: ${acceptM}" \ -H "Accept: ${acceptML}" \ -X DELETE -s "registry.example.org/v2/${repo}/manifests/${digest}", should I also use the command docker exec registry /bin/registry garbage-collect \ /etc/docker/registry/config.yml --delete-untagged afterwards? – Revell

I run this command and I got this: Error: unknown flag: --delete-untagged. I guess its ok to run the entire command whitout this flag? If needed: docker exec registry /bin/registry garbage-collect \ /etc/docker/registry/config.yml – Revell

@Fr0zt That command assumes you're running a current version of the registry:2 image from Hub for your registry. If you're running something else, I don't know what the right command would be. There are lots of registries out there, each are different. – Protect

ok. One more thing, lets say I delete accidentally a digest with curl command. Is there any way to get it back? – Revell

@Fr0zt you have to push it again, there's no undelete. – Protect

I did this process and deleted the manifest then garbage collected. I now get an HTTP/2 404 when I curl it with headers. Great. However, my tag is still there. My pipeline currently is to check for a list of tags, select the oldest one, and then delete its manifest + gc. If the tag persists after this cleanup, then the pipeline perpetually continues to try and remove this tag, except now the manifest returns a 404. Should I run a separate step to clean the tag, or is this indicative of something more fundamental going wrong? – Avigation

@Avigation how did you delete the tag, how are you listing the tags, what was the original media type of the manifest, what accept headers did you use. A minimal reproducible example would be helpful, and that would be best in a separate question. – Protect

I may have figured it out. Ultimately I was able to remove the tags by changing the HEAD to GET, which provided a json with the revision key. Using the delete command above with the revision key's sha256 cleared the entries. Possibly while developing my housekeeping script, I messed up the deletes (I forgot the headers initially), and something got corrupted to where only the revisions SHAs worked. I wasn't able to repush the same images anymore, either, so I ended up deleting the entire registry and starting from scratch anyways. Now it seems to work. Thanks for the great post. – Avigation

Answer 2

There are 2 steps required:

• Deleting the tags
• Running Garbage Collection to free up space

Tag Deletion

There are cleaner ways of doing this with the HTTP/REST API but you can execute a controlled deletion of old tags (>30days) with this command:

find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/tags -type d -mtime +30 -maxdepth 1 -exec rm -rf {} \;

This will effectively "untag" the images. I highly recommend you run the find command without -exec rm first to be sure of what you're deleting!

find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/tags -type d -mtime +30 -maxdepth 1

Kubernetes example (run from host machine):

kubectl exec -it deploy/registry-docker-registry -- \  
  sh -c 'find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/tags -type d -mtime +30 -maxdepth 1 -exec rm -rf {} \;'

Garbage Collection

Finally, run the garbage-collection -m executable.

/bin/registry garbage-collect -m /etc/docker/registry/config.yml

Or, from the host machine you can run it inside the container as follows (Kubernetes Example:

kubectl exec -it deploy/registry-docker-registry -- \  
  /bin/registry garbage-collect -m /etc/docker/registry/config.yml

Fleuron

Only -maxdepth 1 selects the /tags directory too, you should use -mindepth 1 as well. – Zelma

find /var/lib/registry/docker/registry/v2/repositories/*/_manifests/tags considers only one level in the filesystem tree, so it will not match images with / in their name. Better is to search with regex: find /var/lib/registry/docker/registry/v2/repositories/ -regex ".*/tags/[^/]*" -type d -mtime +30 – Zelma

Answer 3

While accepted answer is a good end solution, there is a simpler way to achieve this, without using the API: just delete the tag dirs manually from registry storage and then run the garbage-collect.

First identify where your registry stores the data. By default it goes to /var/lib/registry dir inside the container, but its probably binded to your host one way or the other. I am using docker-compose with data volume so in my case its located at: /var/lib/docker/volumes/registry_data/_data/registry.

You'll find dirs named as your project tags in there, relative to above path: v2/repositories/<repo_name>/_manifests/tags

Just delete the dirs which tag you dont want anymore. For example:

rm -rf /var/lib/docker/volumes/registry_data/_data/registry/v2/repositories/I5/_manifests/tags/T12

Once you are done with that, run the garbage-collect:

docker exec <registry_container_name> registry garbage-collect /etc/docker/registry/config.yml --delete-untagged

Your registry config file may be in some other path, but this is the default. Read more about the garbage collector from docs, but the gist is: the registry binary inside the container includes a garbage-collect command, but you have to run it yourself. Of course you can always let the crontab run it for you.

Burstone

We had no /var/lib/docker/volumes in our registry:2.7.1 but there are "tags" directories inside each repo here /var/lib/registry/docker/registry/v2/repositories/*/_manifests/tags/. – Fleuron

Well as I said, by default it goes to /var/lib/registry, but it depends of your setup. I run registry inside container with a volume, so all registry data is inside the volume. /var/lib/docker/volumes is standard path for docker volumes. – Burstone