So I have a private repository at docker hub and I am trying to download image (blobs) manually using HTTP API.
Now there are some issues
But there is no API in docker HUB api to get list of blobs from a tag and then download it.
There is a docker registry api, but there my username password does not work. What to do?
For an image, you first pull the manifest and parse that for the list of blobs that you need to pull. All of these API's need the same authorization headers used to list the tags. I'm going to be using regctl from my regclient project to query a local registry, but you could also use curl against Hub which I show below.
$ regctl tag ls localhost:5000/library/alpine
3
3-bkup-20210904
3.10
3.11
3.12
3.13
3.14
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
latest
$ regctl manifest get localhost:5000/library/alpine:latest --format body | jq .
{
"manifests": [
{
"digest": "sha256:14b55f5bb845c7b810283290ce057f175de87838be56f49060e941580032c60c",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "amd64",
"os": "linux"
},
"size": 528
},
{
"digest": "sha256:40f396779ba29da16f29f780963bd4ad5b7719e3eb5dec04516d583713256aa8",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm",
"os": "linux",
"variant": "v6"
},
"size": 528
},
{
"digest": "sha256:392d9d85dff31e34d756be33579f05ef493cb1b0edccc36a11b3295365553bfd",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm",
"os": "linux",
"variant": "v7"
},
"size": 528
},
{
"digest": "sha256:4fb53f12d2ec18199f16d7c305a12c54cda68cc622484bfc3b7346a44d5024ac",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm64",
"os": "linux",
"variant": "v8"
},
"size": 528
},
{
"digest": "sha256:e8d9cf28250078f08e890a3466efbefda68a8feac03cc4076d3ada3397370d6e",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "386",
"os": "linux"
},
"size": 528
},
{
"digest": "sha256:d860569a59af627dafee0b0f2b8069e31b07fbdaebe552904dbaec28047ccf64",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "ppc64le",
"os": "linux"
},
"size": 528
},
{
"digest": "sha256:6640b198347e5bf1e9a9dc5fc864e927154275dc31f3d26193b74350a5c94c9f",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "s390x",
"os": "linux"
},
"size": 528
}
],
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"schemaVersion": 2
}
$ regctl manifest get localhost:5000/library/alpine@sha256:14b55f5bb845c7b810283290ce057f175de87838be56f49060e941580032c60c --format body | jq .
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1472,
"digest": "sha256:e9adb5357e84d853cc3eb08cd4d3f9bd6cebdb8a67f0415cc884be7b0202416d"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2812636,
"digest": "sha256:3d243047344378e9b7136d552d48feb7ea8b6fe14ce0990e0cc011d5e369626a"
}
]
}
$ regctl blob get localhost:5000/library/alpine sha256:e9adb5357e84d853cc3eb08cd4d3f9bd6cebdb8a67f0415cc884be7b0202416d | jq .
{
"architecture": "amd64",
"config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh"
],
"Image": "sha256:e211ac20c5c7aaa4ed30d5553654d4679082ec48efcb4d164bac6d50d62653fd",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"container": "b6ba94212561a8075e1d324fb050db160e25035ffcfbbe5b410e411e2b7000e2",
"container_config": {
"Hostname": "b6ba94212561",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"/bin/sh\"]"
],
"Image": "sha256:e211ac20c5c7aaa4ed30d5553654d4679082ec48efcb4d164bac6d50d62653fd",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {}
},
"created": "2022-03-17T04:01:59.188838147Z",
"docker_version": "20.10.12",
"history": [
{
"created": "2022-03-17T04:01:58.883733237Z",
"created_by": "/bin/sh -c #(nop) ADD file:cf4b631a115c2bbfbd81cad2d3041bceb64a8136aac92ba8a63b6c51d60af764 in / "
},
{
"created": "2022-03-17T04:01:59.188838147Z",
"created_by": "/bin/sh -c #(nop) CMD [\"/bin/sh\"]",
"empty_layer": true
}
],
"os": "linux",
"rootfs": {
"type": "layers",
"diff_ids": [
"sha256:5e03d8cae8773cb694fff1d55da34a40d23c2349087ed15ce68476395d33753c"
]
}
}
$ regctl blob get localhost:5000/library/alpine sha256:3d243047344378e9b7136d552d48feb7ea8b6fe14ce0990e0cc011d5e369626a | tar -tvzf - | head
drwxr-xr-x 0/0 0 2022-03-16 16:15 bin/
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/arch -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/ash -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/base64 -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/bbconfig -> /bin/busybox
-rwxr-xr-x 0/0 824984 2022-02-02 13:21 bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/cat -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/chgrp -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/chmod -> /bin/busybox
lrwxrwxrwx 0/0 0 2022-03-16 16:15 bin/chown -> /bin/busybox
...
A few examples if you try doing this with curl:
Get a token (specific to Docker Hub, each registry may have different auth methods and servers):
token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
To use a user/password rather than an anonymous login, call curl with the -u user:pass option:
token=$(curl -u "$user:$pass" -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
| jq -r '.token')
Tags:
curl -H "Authorization: Bearer $token" \
-s "https://registry-1.docker.io/v2/${repo}/tags/list" | jq .
Manifests:
api="application/vnd.docker.distribution.manifest.v2+json"
apil="application/vnd.docker.distribution.manifest.list.v2+json"
curl -H "Accept: ${api}" -H "Accept: ${apil}" \
-H "Authorization: Bearer $token" \
-s "https://registry-1.docker.io/v2/${repo}/manifests/${sha:-$tag}" | jq .
And blobs:
curl -H "Authorization: Bearer $token" \
-s -L -o - "https://registry-1.docker.io/v2/${repo}/blobs/${digest}"
If you want to export everything to be able to later import, regctl image export will convert the image to a tar of manifests and blobs. This is output in the OCI Layout format, and if you only pull a single image manifest (and not a multi-platform manifest) it will also include the needed files for docker load to import the tar.
There is a docker registry api, but there my username password does not work. What to do?
It probably depends on how you supply them. I didn't try it with a private repository on Docker Hub, but here's a script that downloads public images from Docker Hub, and images from private registries:
#!/usr/bin/env bash
set -eu
image=$1
creds=${2-}
# https://github.com/moby/moby/blob/v20.10.18/vendor/github.com/docker/distribution/reference/normalize.go#L29-L57
# https://github.com/moby/moby/blob/v20.10.18/vendor/github.com/docker/distribution/reference/normalize.go#L88-L105
registry=${image%%/*}
if [ "$registry" = "$image" ] \
|| { [ "`expr index "$registry" .:`" = 0 ] && [ "$registry" != localhost ]; }; then
registry=docker.io
else
image=${image#*/}
fi
if [ "$registry" = docker.io ] && [ "`expr index "$image" /`" = 0 ]; then
image=library/$image
fi
if [ "`expr index "$image" :`" = 0 ]; then
tag=latest
else
tag=${image#*:}
image=${image%:*}
fi
if [ "$registry" = docker.io ]; then
registry=https://registry-1.docker.io
elif ! [[ "$registry" =~ ^localhost(:[0-9]+)$ ]]; then
registry=https://$registry
fi
r=`curl -sS "$registry/v2/" \
-o /dev/null \
-w '%{http_code}:%header{www-authenticate}'`
http_code=`echo "$r" | cut -d: -f1`
curl_args=(-sS -H 'Accept: application/vnd.docker.distribution.manifest.v2+json')
if [ "$http_code" = 401 ]; then
if [ "$registry" = https://registry-1.docker.io ]; then
header_www_authenticate=`echo "$r" | cut -d: -f2-`
header_www_authenticate=`echo "$header_www_authenticate" | sed -E 's/^Bearer +//'`
split_into_lines() {
sed -Ee :1 -e 's/^(([^",]|"([^"]|\")*")*),/\1\n/; t1'
}
header_www_authenticate=`echo "$header_www_authenticate" | split_into_lines`
extract_value() {
sed -E 's/^[^=]+="(([^"]|\")*)"$/\1/; s/\\(.)/\1/g'
}
realm=$(echo "$header_www_authenticate" | grep '^realm=' | extract_value)
service=$(echo "$header_www_authenticate" | grep '^service=' | extract_value)
scope=repository:$image:pull
token=`curl -sS "$realm?service=$service&scope=$scope" | jq -r .token`
curl_args+=(-H "Authorization: Bearer $token")
else
curl_args+=(-u "$creds")
fi
fi
manifest=`curl "${curl_args[@]}" "$registry/v2/$image/manifests/$tag"`
config_digest=`echo "$manifest" | jq -r .config.digest`
config=`curl "${curl_args[@]}" -L "$registry/v2/$image/blobs/$config_digest"`
layers=`echo "$manifest" | jq -r '.layers[] | .digest'`
echo "$layers" | \
while IFS= read -r digest; do
curl "${curl_args[@]}" -L "$registry/v2/$image/blobs/$digest" | wc -c
done
Usage:
$ ./download-image.sh hello-world
$ ./download-image.sh library/hello-world
$ ./download-image.sh docker.io/library/hello-world
$ ./download-image.sh myregistry.com/hello-world testuser:testpassword
$ ./download-image.sh localhost:5000/hello-world
Or to be more precise it retrieves the image config and the layers. The thing is, what are you going to do with them? You can try to create a tar archive that looks like the one produced by docker save. Which is what docker-drag basically does. But ideally for that you should know what exactly docker pull and docker save do.
To give you some links to the source code, docker pull (the server part) more or less starts here:
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image.go#L37
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image_routes.go#L78
https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_pull.go#L54
https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_pull.go#L130
https://github.com/moby/moby/blob/v20.10.18/distribution/pull.go#L52
Most of the relevant code is in distribution/pull_v2.go (the high-level part).
The code that does the http requests is in vendor/github.com/docker/distribution/registry/client/auth/session.go and vendor/github.com/docker/distribution/registry/client/repository.go.
docker save:
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image.go#L31
https://github.com/moby/moby/blob/v20.10.18/api/server/router/image/image_routes.go#L160
https://github.com/moby/moby/blob/v20.10.18/daemon/images/image_exporter.go#L16
https://github.com/moby/moby/blob/v20.10.18/image/tarexport/save.go#L187
Most of the code is in image/tarexport/save.go.
But well, apparently docker-drag's developer didn't concern himself with it, and it (docker-drag) seems to work.
In case of private Docker Hub repositories you probably need to add -u user:password to the $realm?service=$service&scope=$scope request.
A couple of remarks about using the API:
user/name (e.g. nginxproxy/nginx-proxy). But with official repositories (that all in fact start with library/) the first segment may be omitted (library/ruby -> ruby). Also there might be just one segment (e.g. in private registries), and occasionaly the first part alone is called a repository (example). Or the second./v2/_catalog route.© 2022 - 2024 — McMap. All rights reserved.