Logging in to private docker registry v2 behind haproxy

Question

I am trying to set up a new Docker Registry (v2) with HAProxy. For the Docker Registry I am using the image from the docker hub and running it with docker run -d -p 5000:5000 -v /path/to/registry:/tmp/registry registry:2.0.1. And this is a subset of my HAProxy configuration:

global
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    tune.ssl.default-dh-param 2048

userlist auth_list
    group docker_registry users root
    user root password ***PASSWORD***

backend docker-registry
    server 127.0.0.1:5000_localhost 127.0.0.1:5000 cookie 127.0.0.1:5000_localhost 

frontend shared-frontend
    mode http
    bind *:80
    bind *:443 ssl crt *** CERT FILES ***
    option accept-invalid-http-request

    acl domain_d.mydomain.com hdr(host) -i d.mydomain.com
    acl auth_docker_registry_root http_auth(auth_list) root
    redirect scheme https if !{ ssl_fc } domain_d.mydomain.com
    http-request auth realm Registry if !auth_docker_registry_root { ssl_fc } domain_d.mydomain.com
    use_backend docker-registry if domain_d.mydomain.com

The important things to note are that I am using HAProxy to do SSL termination and HTTP auth rather than the registry.

My issue occurs when I try to login to the new registry. If I run docker login https://d.mydomain.com/v2/ then enter the user root and password I get the following error messages:

Docker Client:

FATA[0009] Error response from daemon: invalid registry endpoint https://d.mydomain.com/v2/: https://d.mydomain.com/v2/ does not appear to be a v2 registry endpoint. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry d.mydomain.com` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/d.mydomain.com/ca.crt

Docker Daemon:

ERRO[0057] Handler for POST /auth returned error: invalid registry endpoint https://d.mydomain.com/v2/: https://d.mydomain.com/v2/ does not appear to be a v2 registry endpoint. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry d.mydomain.com` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/d.mydomain.com/ca.crt
ERRO[0057] HTTP Error: statusCode=500 invalid registry endpoint https://d.mydomain.com/v2/: https://d.mydomain.com/v2/ does not appear to be a v2 registry endpoint. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry d.mydomain.com` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/d.mydomain.com/ca.crt

So I try adding --insecure-registry d.mydomain.com to:

• /etc/default/docker with DOCKER_OPTS= -H unix:///var/run/docker.sock --insecure-registry d.mydomain.com
• the arguments of starting docker manually with docker -d --insecure-registry d.mydomain.com

neither of these, or any other I have found online, work. Each time, after restarting docker and attempting to log in again gives me the same error message.

A few other things I have tried:

• In a browser going to d.mydomain.com results in a 404
• In a browser going to d.mydomain.com/v2/ results in: {}
• Replacing https://d.mydomain.com/v2/ in the login command with all of these with no success
  • http://d.mydomain.com/v2/
  • d.mydomain.com/v2/
  • http://d.mydomain.com/
  • d.mydomain.com/

This setup with HAProxy doing the SSL termination and HTTP auth has worked in the past using the first version of the registry and older versions of docker. So has anything in Docker registry v2 changed? Does this still work? If it hasn't changed, why won't the --insecure-registry flag do anything anymore?

Also, I have been working on getting this working for a while so I may have forgotten all the things I have tried. If there is something that may work, let me know and I will give it a try.

Thanks, JamesStewy

---

Edit

This edit has been moved to the answer below

Bigeye

what does your backend config look like, I assume you are using nginx to split /v2 out? – Nonagenarian

The config in the question is it. The server points to 127.0.0.1:5000 which is where the docker-registry container is. Unless nginx is built in to the docker registry image then there is no nginx at all (the idea was to use HAProxy as that is what I use for everything else). I am not quite sure what you mean by 'split /v2 out' but if you mean adding reqrep ^([^\ ]*\ /)v2[/]?(.*) \1\2 to the backend to turn http://d.mydomain.com/v2/ to http://d.mydomain.com/ then that didn't appear to work as I just get the same error when I try to login. – Bigeye

Have you made any progress with this? – Myra

Not at this stage. As I alluded to in the post I have been working at this for a while and am currently out of ideas to test. The big problem I have at the moment in trying to progress in that the new docker registry is written in go rather that python so I am having a bit more difficulty trying to understand why it isn't working. – Bigeye

Any news on this one. I am having the same issue and nothing seems to work. If I disable Basic Auth it works fine using HAProxy as https terminator only. – Prognostic

I have updated the original post with my current findings. TL;DR, made some improvements (login works) but it still doesn't work (can't push or pull). – Bigeye

Thanks @Bigeye that was my issue. Need to add rsprep ^Location:\ http://(.*) Location:\ https://\1 to the haproxy backend pass thru options. Using Haproxy behind pfsense. – Adapt

Answer 1

I have got it working. So here is my new config:

haproxy.cfg

global
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    tune.ssl.default-dh-param 2048

userlist auth_list
    group docker_registry users root
    user root password ***PASSWORD***

backend docker-registry
    server 127.0.0.1:5000_localhost 127.0.0.1:5000 cookie 127.0.0.1:5000_localhost 

backend docker-registry-auth
    errorfile 503 /path/to/registry_auth.http

frontend shared-frontend
    mode http
    bind *:80
    bind *:443 ssl crt *** CERT FILES ***
    option accept-invalid-http-request

    acl domain_d.mydomain.com hdr(host) -i d.mydomain.com
    redirect scheme https if !{ ssl_fc } domain_d.mydomain.com
    acl auth_docker_registry_root http_auth(auth_list) root
    use_backend docker-registry-auth if !auth_docker_registry_root { ssl_fc } domain_d.mydomain.com
    rsprep ^Location:\ http://(.*) Location:\ https://\1
    use_backend docker-registry if domain_d.mydomain.com

registry_auth.http

HTTP/1.0 401 Unauthorized
Cache-Control: no-cache
Connection: close
Content-Type: text/html
Docker-Distribution-Api-Version: registry/2.0
WWW-Authenticate: Basic realm="Registry"

<html><body><h1>401 Unauthorized</h1>
You need a valid user and password to access this content.
</body></html>

The differences being the http-request auth line has been replaced with use_backend docker-registry-auth. The backend docker-registry-auth has no servers to it will always give a 503 error. But the 503 error file has been changed to registry_auth.http. In registry_auth.http the error code is overridden to 401, the header WWW-Authenticate is set to Basic realm="Registry", the basic HAProxy 401 error page is supplied and, most importantly, the header Docker-Distribution-Api-Version is set to registry/2.0.

As a result this hacky work-around setup works exactly the same as the old http-request auth line except the custom header Docker-Distribution-Api-Version is now set. This allows this set up to pass the test which starts on line 236 of https://github.com/docker/docker/blob/v1.7.0/registry/endpoint.go.

So now when I run docker login d.mydomain.com, login is successful and my credentials are added to .docker/config.json.

The second issue was that I couldn't push to the new repository even through it logged in. This was fixed by adding the rsprep line in the frontend. What this line does is modify the Location header (if it exists) to turn all http:// to https://.

I also found this bit of documentation for future reference.

Bigeye

Answer 2

As a small clarification to the previous answer: I had to change this line:

WWW-Authenticate: Basic realm="Registry"

To this:

WWW-Authenticate: Basic realm="Registry realm"

and then everything worked...

BTW, hashing the pass can be done using mkpasswd (part of whois deb package)

Culmiferous