Blob Unknown when pushing to custom registry through apache proxy

S

2

11

I have a custom docker registry running in a container bound to localhost like this 127.0.0.1:5010->5000/tcp. When I push an image to it locally it works. When I try to push an image to it remotely it connects and starts to push the image and then I get err.code="blob unknown". The registry is proxied through Apache with ssl enabled in Apache. TLS isn't on in the registry since it's bound to localhost it really doesn't need it. Not sure what's broken here, any suggestions?

Command (after docker login was successful):

ubuntu@ip-172-31-31-137  ~  docker push registry.sniftershifter.com/nginx
The push refers to repository [registry.sniftershifter.com/nginx]
f12c6cf07176: Pushing [==================================================>]  3.584kB
341dde1390a8: Pushing [===================>                               ]  20.87MB/53.68MB
9c46f426bcb7: Pushing [====>                                              ]  5.394MB/55.29MB
unknown blob

docker-compose.yml

registry:
  container_name: registry
  restart: always
  image: registry:2
  ports:
    - 127.0.0.1:5010:5000
  environment:
    REGISTRY_AUTH: htpasswd
    REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
    REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
  volumes:
    - /home/jrow/docker_registry/data:/var/lib/registry
    - /home/jrow/docker_registry/auth:/auth

Apache config:

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerAdmin [email protected]
    ServerName registry.sniftershifter.com
    ProxyPreserveHost On
    # setup the proxy
    <Proxy *>
        Order allow,deny
        Allow from all
    </Proxy>
    SetEnv proxy-initial-not-pooled 1
    # SSLProxyEngine on
    ProxyPass / http://localhost:5010/ KeepAlive=On Timeout=600
    ProxyPassReverse / http://localhost:5010/
SSLCertificateFile /etc/letsencrypt/live/registry.sniftershifter.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/registry.sniftershifter.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

Log of attempt from registry:

172.17.0.1 - - [24/Jul/2018:21:43:18 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=error msg="response completed with error" auth.user.name=jeremiah err.code="blob unknown" err.detail=sha256:683abbb4ea60e108164f1d351e7bcf13daf45941137d800086447874df05f48e err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=96767d96-68e9-4d53-9f9e-970fabb5f94b http.request.method=HEAD http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/sha256:683abbb4ea60e108164f1d351e7bcf13daf45941137d800086447874df05f48e" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=11.698327ms http.response.status=404 http.response.written=157 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 vars.digest="sha256:683abbb4ea60e108164f1d351e7bcf13daf45941137d800086447874df05f48e" vars.name=nginx version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "HEAD /v2/nginx/blobs/sha256:683abbb4ea60e108164f1d351e7bcf13daf45941137d800086447874df05f48e HTTP/1.1" 404 157 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=error msg="response completed with error" auth.user.name=jeremiah err.code="blob unknown" err.detail=sha256:a58abb4a79903b460a6db8a1237fd38c67adfd5a997a818e7b5f70d29032c256 err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=7613fa79-8969-4bf6-97b3-78b989306c94 http.request.method=HEAD http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/sha256:a58abb4a79903b460a6db8a1237fd38c67adfd5a997a818e7b5f70d29032c256" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=13.077594ms http.response.status=404 http.response.written=157 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 vars.digest="sha256:a58abb4a79903b460a6db8a1237fd38c67adfd5a997a818e7b5f70d29032c256" vars.name=nginx version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "HEAD /v2/nginx/blobs/sha256:a58abb4a79903b460a6db8a1237fd38c67adfd5a997a818e7b5f70d29032c256 HTTP/1.1" 404 157 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=error msg="response completed with error" auth.user.name=jeremiah err.code="blob unknown" err.detail=sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574 err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=f3ccafd2-9202-4013-8689-2fdc95b95c6a http.request.method=HEAD http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=11.329439ms http.response.status=404 http.response.written=157 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 vars.digest="sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" vars.name=nginx version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "HEAD /v2/nginx/blobs/sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574 HTTP/1.1" 404 157 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=info msg="response completed" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=f76a8ffd-a554-4214-b98e-290031a9dc89 http.request.method=POST http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=35.031305ms http.response.status=202 http.response.written=0 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=info msg="response completed" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=047fe1c3-86fe-44d3-87e7-169ca1d481a6 http.request.method=POST http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=29.598379ms http.response.status=202 http.response.written=0 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=info msg="response completed" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=18639e66-864a-4e00-b72e-9815e3d0382b http.request.method=POST http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/uploads/" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=36.164985ms http.response.status=202 http.response.written=0 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "POST /v2/nginx/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:19Z" level=info msg="response completed" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=badceb1d-cc11-4add-928d-c63b50dd669d http.request.method=GET http.request.referer="http://registry.sniftershifter.com/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=Ov8UC8vNnZ--PM57I7u54dQ4MqQMrn1gAa-ly_wJloF7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTkuMTU0MzExNzU5WiJ9" http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=Ov8UC8vNnZ--PM57I7u54dQ4MqQMrn1gAa-ly_wJloF7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTkuMTU0MzExNzU5WiJ9" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=24.116927ms http.response.status=204 http.response.written=0 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:19 +0000] "GET /v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=Ov8UC8vNnZ--PM57I7u54dQ4MqQMrn1gAa-ly_wJloF7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTkuMTU0MzExNzU5WiJ9 HTTP/1.1" 204 0 "http://registry.sniftershifter.com/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=Ov8UC8vNnZ--PM57I7u54dQ4MqQMrn1gAa-ly_wJloF7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTkuMTU0MzExNzU5WiJ9" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:20Z" level=info msg="response completed" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=5b51b755-1794-41e7-a634-4cab9669e5a5 http.request.method=GET http.request.referer="http://registry.sniftershifter.com/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=fD-oMCYtp_KhN3xa_8dFxdGgBmz_NS6US-Y8irYgFAx7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTlaIn0%3D&digest=sha256%3Ab43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=fD-oMCYtp_KhN3xa_8dFxdGgBmz_NS6US-Y8irYgFAx7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTlaIn0%3D&digest=sha256%3Ab43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.duration=19.471627ms http.response.status=204 http.response.written=0 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:20 +0000] "GET /v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=fD-oMCYtp_KhN3xa_8dFxdGgBmz_NS6US-Y8irYgFAx7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTlaIn0%3D&digest=sha256%3Ab43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574 HTTP/1.1" 204 0 "http://registry.sniftershifter.com/v2/nginx/blobs/uploads/7eb09801-1fb6-4d55-97bd-e64ab6ff0bd8?_state=fD-oMCYtp_KhN3xa_8dFxdGgBmz_NS6US-Y8irYgFAx7Ik5hbWUiOiJuZ2lueCIsIlVVSUQiOiI3ZWIwOTgwMS0xZmI2LTRkNTUtOTdiZC1lNjRhYjZmZjBiZDgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTgtMDctMjRUMjE6NDM6MTlaIn0%3D&digest=sha256%3Ab43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"
time="2018-07-24T21:43:20Z" level=error msg="response completed with error" auth.user.name=jeremiah err.code="blob unknown" err.detail=sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574 err.message="blob unknown to registry" go.version=go1.7.6 http.request.host=registry.sniftershifter.com http.request.id=076e9eaa-2b0a-40c6-820a-3bc3180844c3 http.request.method=HEAD http.request.remoteaddr=52.22.187.80 http.request.uri="/v2/nginx/blobs/sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" http.request.useragent="docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=12.122095ms http.response.status=404 http.response.written=157 instance.id=8d9bc71b-b767-46e3-8acd-37cb676bb4d7 vars.digest="sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574" vars.name=nginx version=v2.6.2

172.17.0.1 - - [24/Jul/2018:21:43:20 +0000] "HEAD /v2/nginx/blobs/sha256:b43279c1d51c22a69660908e3df2bff2cb6ccd35a2e07bafd47964517ac01574 HTTP/1.1" 404 157 "" "docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f40 kernel/4.4.0-1061-aws os/linux arch/amd64 UpstreamClient(Docker-Client/18.03.1-ce \\(linux\\))"

Stomy answered 24/7, 2018 at 22:2 Comment(0)

S

20

Adding these lines to the apache config for this vhost fixed it:

Header add X-Forwarded-Proto "https"
RequestHeader add X-Forwarded-Proto "https"

Stomy answered 31/7, 2018 at 16:58 Comment(5)

this works like a charm i was using nginx in front fixed it with proxy_set_header X-Forwarded-Proto https; – Regenerative 12/2, 2019 at 8:48

I had to do this when using traefik as well, where the https portion was handled before traefik. Use 'traefik.frontend.headers.customRequestHeaders=X-Forwarded-Proto:https' if using docker swarm labels for instance. – Aaren 11/6, 2019 at 19:29

does anyone know why this is necessary? – Chirr 11/12, 2019 at 14:19

just a thing to have in mind, apache module headers needs to be enabled. I had to do a2enmod headers on my machine first – Anthropology 16/2, 2020 at 17:23

Thank you i spent many hours but i finally it is fixed with above comment. May i ask what are those doing actually – Incorrupt 2/7, 2020 at 20:33

D

1

Add the public host (i.e. URL of the host, https://myhost.com in the example below) to the registry's configuration, i.e. add the http.host in the yml configuration, or when configuring via environment variables, specify it via the REGISTRY_HTTP_HOST environment variable like so:

docker run -e REGISTRY_HTTP_HOST=https://myhost.com -d -p 127.0.0.1:5000:5000 --restart=always --name registry registry:2

Diogenes answered 28/4, 2023 at 11:54 Comment(0)

Recommended topics

Hot tags