Docker environment via nginx proxy results in 502 Bad Gateway

Question

I am using nginx-proxy to connect subdomains to different docker services. Since one of the last image updates I am not able to connect to my gitlab instance anymore. I don't know if a nginx-proxy or gitlab update resulted in this issue. When I try to connect to gitlab.mydomain.com I get the following errors:

1. Browser: 502 Bad Gateway. nginx/1.13.3

2. nginx-proxy logs:

   nginx-proxy_1 | nginx.1 | 2017/08/14 11:44:10 [error] 39#39: *1672 connect() failed (111: Connection refused) while connecting to upstream, client: 178.201.120.94, server: gitlab.mydomain.com, request: "GET / HTTP/2.0", upstream: "https://172.18.0.2:443/", host: "gitlab.mydomain.com"

   nginx-proxy_1 | nginx.1 | gitlab.mydomain.com 178.201.120.94 - - [14/Aug/2017:11:44:10 +0000] "GET / HTTP/2.0" 502 575 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"

   nginx-proxy_1 | nginx.1 | 2017/08/14 11:44:11 [error] 39#39: *1672 connect() failed (111: Connection refused) while connecting to upstream, client: 178.201.120.94, server: gitlab.mydomain.com, request: "GET /favicon.ico HTTP/2.0", upstream: "https://172.18.0.2:443/favicon.ico", host: "gitlab.mydomain.com", referrer: "https://gitlab.***.com/"

My docker-compose file:

version: '2'

services:
   nginx-proxy:
     image: jwilder/nginx-proxy
     ports:
     - "80:80"
     - "443:443"
     volumes:
     - /var/local/nginx/certs:/etc/nginx/certs
     - /etc/letsencrypt:/etc/letsencrypt
     - /var/run/docker.sock:/tmp/docker.sock:ro

   gitlab:
     image: gitlab/gitlab-ce:latest
     ports:
     - "2222:22"
     hostname: 'gitlab.***.com'
     expose:
     - 443
     - 22
     - 80
     - 25
     environment:
     - "VIRTUAL_HOST=gitlab.***.com,www.gitlab.***.com"
     - "VIRTUAL_PORT=443"
     - "VIRTUAL_PROTO=https"
     volumes:
     - "gitlab-config:/etc/gitlab"
     - "gitlab-log:/var/log/gitlab"
     - "gitlab-data:/var/opt/gitlab"
     - "/etc/letsencrypt:/etc/letsencrypt"

Any ideas on that? Thanks.
Edit: Networksettings from docker inspect gitlab:

    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "...",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {
            "22/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "2222"
                }
            ],
            "25/tcp": null,
            "443/tcp": null,
            "80/tcp": null
        },
        "SandboxKey": "/var/run/docker/netns/...",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "docker_default": {
                "IPAMConfig": null,
                "Links": null,
                "Aliases": [
                    "abb170528bcc",
                    "gitlab"
                ],
                "NetworkID": "...",
                "EndpointID": "...",
                "Gateway": "172.18.0.1",
                "IPAddress": "172.18.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:12:00:02"
            }
        }
    }

Sosanna

Do docker ps and docker inspect on your gitlab container and see if the IP it is fetching is correct or not? – Cenotaph

a docker inspect on gitlab gave me the following values: "Gateway": "172.18.0.1", "IPAddress": "172.18.0.2", ... looks like an error to me? – Sosanna

Check how many networks are associated with gitlab in your case? – Cenotaph

i have added the networksettings part from docker inspect. is this sufficient for you? is this error maybe related to SSL encryption? or some ipv6 stuff? – Sosanna

In order to make your docker-compose.yml testable by someone else, it would be useful to have the contents of your gitlab-config volume. Can you post a version stripped of your private data? – Photoemission

Did you configure unicorn properly? Unicorn in gitlab starts up slowly, it could be the reason. Another one is port availability. – Loveridge

I dont think I have configured anything concerning unicorn. In fact, this is the first time I hear about it. However, I also tried to connect to gitlab some hours after starting the image. – Sosanna

What about port availability? How can I debug it? – Sosanna

Answer 1

this setup works for me, maybe you can compare it and find out, what's wrong:

https://superuser.com/a/1242682/762931

maybe it is this: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1307

Ivonne

thank you, I will try it out. If it works I will post the necessary modifications and accept your answer! – Sosanna

Are there any files within /etc/gitlab/trusted-certs on your gitlab docker machine? I feel like there are some problems concerning the SSL certificates. – Sosanna

No there aren't, they are being put into the volume and then directly into the /etc/gitlab/ssl folder in the container – Ivonne