Check if image:tag combination already exists on docker hub

Question

As part of a bash script, I want to check if a particularly docker image:tag combination exists on docker hub. Also, it will be a private repository.

i.e. the pseudocode would be like:

tag = something
if image:tag already exists on docker hub:
    Do nothing
else
    Build and push docker image with that tag

Aneroid

I believe we have the same issue. Not sure if this is relevant to you, but we're running our own Docker Registry, and it exposes an API that you could use. docs.docker.com/registry/spec/api/#listing-image-tags I'll probably add a step in my CI build that queries the API so that it doesn't overwrite an existing tag. – Glennaglennie

Im surprised this is not part of the docker repository API.... – Flinger

One could use dip to check whether an image resides in a docker-registry. I release version 1.0.0 today. – Gametophyte

Does anyone here know where would be the right place to ask for this feature in Docker? I would like to send a request for this feature to the Docker team (or understand why this won't be a good candidate be inside Docker). – Sanitize

Wooohoo, 100 upvote, I nailed it)) – Giliane

Answer 1

Please try this one

function docker_tag_exists() {
    curl --silent -f -lSL https://index.docker.io/v1/repositories/$1/tags/$2 > /dev/null
}

if docker_tag_exists library/nginx 1.7.5; then
    echo exist
else 
    echo not exists
fi

---

Update:

In case of usage Docker Registry v2 (based on this):

# set username and password
UNAME="user"
UPASS="password"

function docker_tag_exists() {
    TOKEN=$(curl -s -H "Content-Type: application/json" -X POST -d '{"username": "'${UNAME}'", "password": "'${UPASS}'"}' https://hub.docker.com/v2/users/login/ | jq -r .token)
    curl --silent -f --head -lL https://hub.docker.com/v2/repositories/$1/tags/$2/ > /dev/null
}

if docker_tag_exists library/nginx 1.7.5; then
    echo exist
else 
    echo not exists
fi

Wixted

what about non-docker.io registries? – Strenta

@WayNo what do you use? Please share a link to the documentation. – Wixted

Amazon ECR. docs.aws.amazon.com/AmazonECR/latest/userguide/… . Instead of docker.io it usually follows aws_account_id.dkr.ecr.region.amazonaws.com in its URL naming convention, but not sure if simply replacing the domain name would get the corresponding result. – Strenta

@WayNo I've updated the post with the function for Docker Registry v2. I believe, to use this function with non-docker.io registries you should do the next: use UNAME and UPASS acquired from aws ecr get-login command; and aws_account_id.dkr.ecr.region.amazonaws.com as host. – Wixted

Here's a script for AWS ECR: gist.github.com/1ec2102c8c0d48592750b1f0b5306cc9 – Girlfriend

This would work for ECR: aws ecr describe-images --repository-name <repo> --image-ids imageTag=latest This will remove the need for crafting any curl/authorization. – Helsinki

For AWS, this worked for me: docs.aws.amazon.com/AmazonECR/latest/userguide/… – Adjustment

on v2, to avoid parsing a paginated list of tags just make a HEAD request for the one you care about: curl --silent -f --head -lSL https://registry.example.com/v2/$1/manifests/$2 > /dev/null – Jolley

Tested this with a few repos I'm currently working with, the page size parameter stops working past ~1400 as far as I can tell. Why do these repos have that many tags you ask? Well .... thats a good question. – Rhone

Thanks, @KyleJones! I've improved the script by removing the paginated response. – Wixted

For PowerShell: gist.github.com/webbertakken/1789a4683a99e2a62b975ff436a85382 – Trubow

The v1 docker.io endpoint was recently deprecated, and a script I was using to check tag existence stopped working. I switched from the URL in this answer to https://hub.docker.com/v2/repositories/library/{image}/tags/{tag} for official docker images, eg. python:3, and https://hub.docker.com/v2/namespaces/{namespace}/repositories/{image}/tags/{tag} for images with namespaces, eg. foo/bar:1.0. – Enyedy

Answer 2

Update: Docker-Free solution see below

Using Docker

This is the solution I use with gitlab using the docker:stable image.

Login

docker login -u $USER -p $PASSWORD $REGISTRY

Check whether it's there:

docker manifest inspect $IMGNAME:$IMGTAG > /dev/null ; echo $?

docker will return 0 on success or 1 on failure.

If you get a warning: Update Docker or enable experimental client-features:

Set the environment variable DOCKER_CLI_EXPERIMENTAL to enabled (See Matěj's answer below)

Alternatively adjust the config (original answer):

echo '{"experimental": "enabled"}' > ~/.docker/config.json

This will also overwrite your config. If that is not an option you need to do that manually or use jq, sed or whatever you have available.

Testing without Docker

If you don't have access to a docker-daemon you can give Skopeo or crane a try.

Novgorod

Shame this is still experimental – Unemployable

actually your first command will overwrite the whole config file, so it's worth adding this string by editing the file – Talmudist

One-liner for the lazy: jq '. + {"experimental": "enabled"}' < ~/.docker/config.json | sponge ~/.docker/config.json – Genital

As mentioned in a different answer below, you can avoid changing your docker config file by using DOCKER_CLI_EXPERIMENTAL=enabled – Multistage

@MarkTickner more shame it still doesn't work (I get manifest not found, while pull fetches the image). – Jessamine

Why is this is still experimental 2 years later? – Lauren

it seems non-experimental already. I freshly installed docker, and without enabling any experimental stuff it works for me. – Coenocyte

On my system the docker manifest inspect command may hang as long as 1 minute and 36 seconds if I try to inspect an image that does not exist (using JFrog Artifactory as remote registry). – Rachaba

@Rachaba This seems to be a issue with Artifactory, or even your Artifactory instance. The Server is supposed to answer with an appropriate error-code: docs.docker.com/engine/api/v1.41 Thus, for some reason your server seems to thoroughly ponder its answer. Maybe some 2FA-issue or proxy-function that waits for a secondary server until it times out. – Novgorod

Notice that this doesn't work if you're building cross-platform manifests with docker manifest create -- you'll just always get the manifest you've just created. – Brough

I guess that the answer is right but you should replace the > to 2> in order to redirect the stderr to /dev/null: bash docker manifest inspect $IMGNAME:$IMGTAG 2> /dev/null ; echo $? – Rigmarole

@xserrat: Why would I want to do that? If there is a problem connecting the server I want to see that in the logs. Also the error-messages are quite short (stderr), while the whole manifest (stdout) is rather long ans spams my log. – Novgorod

@igagis: According to Docker docs, it's still experimental (docs.docker.com/engine/reference/commandline/manifest), but from Docker 20.10, experimental features are enabled by default (docs.docker.com/engine/reference/commandline/cli/…). That's likely why it works for you without enabling anything. – Pronucleus

Answer 3

To build on morty's answer, notice that docker supports setting the experimental flag using environment variable:

DOCKER_CLI_EXPERIMENTAL Enable experimental features for the cli (e.g. enabled or disabled)

The snippet therefore becomes:

tag=something
if DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect $image:$tag >/dev/null; then
    Do nothing
else
    Build and push docker image with that tag
fi

Carpogonium

I'm using docker cli Docker version 20.10.7 and it seems that you don't need to enabled DOCKER_CLI_EXPERIMENTAL any more – Robinia

Answer 4

Easiest:

docker pull alpine:invalid > /dev/null && echo "success" || echo "failed"

Pulls & prints success if image exists, or prints failed if it doesn't:

You can even export it in a var if using in bash script:

Note that this will pull the image if it exists. Beware of the overhead cost before using this solution.

Bibliolatry

Note that this will pull the container if it exists. Pulling even the small alpine is quite an overhead just to get that information. Not mentioning the several GB-Image one often has in CI-Environments. – Novgorod

True. I couldn't agree more. I added this info in this answer. Thanks for the input. – Bibliolatry

This is perfect for my AWS CodeBuild layer caching. I just want to attempt to yank it and I don't care if it exists or not. It'll get built and cached anyway – Onassis

If the expectation is that an image does not exist, then it's no problem. Furthermore, in the unlikely event that the image does exist, like when you accidentally built a release using an old tag, this would automatically overwrite your new image back to what it was before, which is quite useful. – Interlunation

Answer 5

I have a docker private repo stood up on my LAN using registry:2, private CA, and basic auth.

I just looked at the official docker API docs (https://docs.docker.com/registry/spec/api/) and came up with this solution which seems pretty elegant, easy to debug, customize, and is CICD/scripting friendly.

curl --silent -i -u "demoadmin":"demopassword" https://mydockerrepo.local:5000/v2/rancher/pause/manifests/3.1 | grep "200 OK"

--silient gets rid of some extra text
-i is what makes the return code "200 OK" show up

if it exists return code is 0, if doesn't exist return code is 1 you can verify that using
Bash# echo $?

Champignon

I wanted to know how to check if a tag exisst in a private repo via rest without doing /v2/imagename/tags/list and this is the perfect answer! For some reason it was really hard to find this answer on stackoverflow. – Wigwag

Answer 6

Here's a Bash function that will help:

docker_image_exists() {
  local image_full_name="$1"; shift
  local wait_time="${1:-5}"
  local search_term='Pulling|is up to date|not found'
  local result="$((timeout --preserve-status "$wait_time" docker 2>&1 pull "$image_full_name" &) | grep -v 'Pulling repository' | egrep -o "$search_term")"
  test "$result" || { echo "Timed out too soon. Try using a wait_time greater than $wait_time..."; return 1 ;}
  echo $result | grep -vq 'not found'
}

Usage example:

docker_image_exists elifarley/docker-dev-env:alpine-sshd && \
  echo EXISTS || \
  echo "Image does not exist"

Acolyte

That looks like it checks for the image's existence by trying to pull it, which is overkill - and if it succeeds, will change the images on the local host. – Thrombo

It does try to pull it, but the script aborts the operation if pulling the image takes more than the selected timeout value (5 seconds by default). – Acolyte

Ah, fair enough. Might be useful to add a bit of explanation to the answer, going through what it does and doesn't do? – Thrombo

could someone write a bash script that uses the api instead? – Representation

After running this, it looks like docker pull on an image breaks association of that image with all running containers of that image. Be cautious, this function will cause side effects on your running containers! – Strenta

Answer 7

Just a small improvement of Evgeny Oskin's solution. When it comes to a user repo that hasn't been created yet, jq says that it "Cannot iterate over null". To overcome it. one can skip not present blocks with ? Here is a modification to above mentioned solution that is applicable to a public repo in particular:

#!/usr/bin/env bash

function docker_image_tag_exists() {
    EXISTS=$(curl -s https://hub.docker.com/v2/repositories/$1/tags/?page_size=10000 | jq -r "[.results? | .[]? | .name == \"$2\"] | any")
    test ${EXISTS} = true
}

if docker_image_tag_exists $1 $2; then
    echo "true"
else
    echo "false"
fi

Zak

Answer 8

I've been struggling with this for some time, and I've arrived at this solution:

docker manifest inspect $registry/$imageName:$tag > /dev/null 2>&1 && echo yes || echo no

This outputs 'yes' if the image already exists and 'no' if it does not.

This does not pull the image. I've also tried 'docker image inspect' but starting getting an error so switched to this approach instead.

Hope it helps!

Radome

Answer 9

I was struggling getting this to work for a private docker hub repository and finally decided to write a ruby script instead, which works as of today. Feel free to use!

#!/usr/bin/env ruby
require 'base64'
require 'net/http'
require 'uri'

def docker_tag_exists? repo, tag
  auth_string = Base64.strict_encode64 "#{ENV['DOCKER_USER']}:#{ENV['DOCKER_PASSWORD']}"
  uri = URI.parse("https://registry.hub.docker.com/v1/repositories/#{repo}/tags/#{tag}")
  request = Net::HTTP::Get.new(uri)
  request['Authorization'] = "Basic #{auth_string}"
  request['Accept'] = 'application/json'
  request['Content-Type'] = 'application/json'
  response = Net::HTTP.start(request.uri.hostname, request.uri.port, use_ssl: true) do |http|
    http.request(request)
  end
  (response.body == 'Tag not found') ? 0 : 1
end

exit docker_tag_exists? ARGV[0], ARGV[1]

Note: you need to specify DOCKER_USER and DOCKER_PASSWORD when calling this like...

DOCKER_USER=XXX DOCKER_PASSWORD=XXX config/docker/docker_hub.rb "NAMESPACE/REPO" "TAG" && echo 'latest'

This line would print out 'latest', if authentication is successful and the specified tag does not exists! I was using this in my Vagrantfile when trying to fetch a tag based on the current git branch:

git rev-parse --symbolic-full-name --abbrev-ref HEAD

Consonantal

Answer 10

All of the options above assume that you can authenticate using username/password. There are a lot of cases where this is inconvenient, for example when using Google Container Registry, for which one would run gcloud auth configure-docker gcr.io first. That command installs an authentication helper for Docker, and you wouldn't want to manage that token yourself.

One tool that supports these docker authentication helpers, and also allows getting a manifest - like experimental Docker - is crane.

Example using crane:

# you would have done this already
gcloud auth configure-docker gcr.io;

# ensure we have crane installed
which crane || (echo 'installing crane' && GO111MODULE=on go get -u github.com/google/go-containerregistry/cmd/crane)

# check manifest
crane manifest ubuntu || echo "does not exist"

Snips

Answer 11

If you are querying Hub for the existence of a tag, make sure you use a HEAD rather than a GET request. The GET request count against your rate limit. A script that does this specific to Docker Hub, only supports the Docker media types, and anonymous logins that are rate limited by your requesting IP, looks like:

$ more ~/data/docker/registry-api/manifest-v2-head.sh 
#!/bin/sh

ref="${1:-library/ubuntu:latest}"
sha="${ref#*@}"
if [ "$sha" = "$ref" ]; then
  sha=""
fi
wosha="${ref%%@*}"
repo="${wosha%:*}"
tag="${wosha##*:}"
if [ "$tag" = "$wosha" ]; then
  tag="latest"
fi
api="application/vnd.docker.distribution.manifest.v2+json"
apil="application/vnd.docker.distribution.manifest.list.v2+json"
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" 
\
        | jq -r '.token')
curl -H "Accept: ${api}" -H "Accept: ${apil}" \
     -H "Authorization: Bearer $token" \
     -I -s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}" 

To work on other registries, handle more media types (like the OCI types), and handle logins, use a tool like crane, skopeo, or my own regclient:

# the "image digest" command uses a HEAD instead of a GET
if regctl image digest registry.example.com/repo:tag >/dev/null 2>&1; then
  echo tag exists
else
  echo tag does not exist
fi

Baudoin

Answer 12

For the local docker registry, you can try this:

function docker_tag_exists() {
    curl --silent -f -lSL http://localhost:5000/v2/$1/manifests/$2 > /dev/null
}
if docker_tag_exists image_on_local latest; then
    echo exists
else
    echo not exists
fi

Crosslegged

Answer 13

I like solutions based on docker.

This oneliner is what I use in our CI:

 docker run --rm anoxis/registry-cli -l user:password -r registry-url -i docker-name | grep -q docker-tag || echo do something if not found

Terrance

You should elaborate more on what the various parts of your command are. It looks like it would be a sweet command, however, the -i docker-name portion in particular is tripping me up and I have no clue what to put there. – Caudell