Traefik > "Bad gateway" (error 502) for some containers

Question

I meet some problems using traefik with docker and I don't know why.

With some containers, it's works like a charm and for other ones, I have an error when I try to access to these ones : Bad gateway (error 502).

Here is my traefik.toml :

# Service logs (here debug mode)
debug = true
logLevel = "DEBUG"

defaultEntryPoints = ["http", "https"]

# Access log
filePath = "/var/log/traefik/access.log"
format = "common"

################################################################
# Web configuration backend
################################################################
[web]
address = ":8080"

################################################################
# Entry-points configuration
################################################################
[entryPoints]
  [entryPoints.http]
    address = ":80"
    [entryPoints.http.redirect]
      entryPoint = "https"
  [entryPoints.https]
    address = ":443"
    [entryPoints.https.tls]

################################################################
# Docker configuration backend
################################################################
[docker]
domain = "domain.tld"
watch = true
exposedbydefault = false
endpoint = "unix:///var/run/docker.sock"

################################################################
# Let's encrypt
################################################################
[acme]
email = "[email protected]"
storageFile = "acme.json"
onDemand = false
onHostRule = true
entryPoint = "https"

[acme.httpChallenge]
  entryPoint = "http"

[[acme.domains]]
  main = "domain.tld"
  sans = ["docker.domain.tld", "traefik.domain.tld", "phpmyadmin.domain.tld", "perso.domain.tld", "muximux.domain.tld", "wekan.domain.tld", "wiki.domain.tld", "cloud.domain.tld", "email.domain.tld"]

Here is my docker-compose.yml (for portainer, which is a container which works) :

version: '2'

services:
  portainer:
    restart: always
    image: portainer/portainer:latest
    container_name: "portainer"
#Automatically choose 'Manage the Docker instance where Portainer is running' by adding <--host=unix:///var/run/docker.sock> to the command
    ports:
      - "9000:9000"
    networks:
      - traefik-network
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ../portainer:/data
    labels:
      - traefik.enable=true
      - traefik.backend=portainer
      - traefik.frontend.rule=Host:docker.domain.tld
      - traefik.docker.network=traefik-network
      - traefik.port=9000
      - traefik.default.protocol=http

networks:
  traefik-network:
    external : true

If I go to docker.domain.tld, it works ! and in https, with valide let's encrypt certificate :)

Here is my docker-compose.yml (for dokuwiki, which is a container which does not work) :

version: '2'

services:
  dokuwiki:
    container_name: "dokuwiki"
    image: bitnami/dokuwiki:latest
    restart: always
    volumes:
      - ../dokuwiki/data:/bitnami
    ports:
      - "8085:80"
      - "7443:443"
    networks:
      - traefik-network
    labels:
      - traefik.backend=dokuwiki
      - traefik.docker.network=traefik-network
      - traefik.frontend.rule=Host:wiki.domain.tld
      - traefik.enable=true
      - traefik.port=8085
      - traefik.default.protocol=http

networks:
  traefik-network:
    external: true

If I go to wiki.domain.tld, it does not work ! I have a bad gateway error on the browser. I have tried to change the traefik.port to 7443 and the traefik.default.protocol to https but I have the same error. Of course it works when I try to access the wiki with the IP and the port (in http / https). I have bad gateway only when I type wiki.domain.tld.

So, I don't understand why it works for some containers and not for other ones with the same declaration.

Lorenzo

Answer 1

The traefik port should be the http port of the container, not the published port on the host. It communicates over the docker network, so publishing the port is unnecessary and against the goals of only having a single port published with a reverse proxy to access all the containers.

In short, you need:

traefik.port=80

---

Since this question has gotten lots of views, the other reason lots of people see a 502 from traefik is placing the containers on a different docker network from the traefik instance, or having a container on multiple networks and not telling traefik which network to use. This doesn't apply in your case since you have the following lines in your compose file that match up with the traefik service's network:

services:
  dokuwiki:
    networks:
      - traefik-network
    labels:
      - traefik.docker.network=traefik-network
networks:
  traefik-network:
    external : true

Even if you only assign a service to a single network, some actions like publishing a port will result in your service being attached to two different networks (the ingress network being the second). The network name in the label needs to be the external name, which in your case is the same, but for others that do not specify their network as external, it may have a project or stack name prefixed which you can see in the docker network ls output.

Br

Ohhh great BMitch, it works ! Thanks a lot :) I have misunderstood this variable ! – Lorenzo

Another little question : If I put the http port, it works. But if I put the https port (443) in traefik.port and https in traefik.default.protocol, I get an "Internal Server Error". Do you know why ?? – Lorenzo

@Lorenzo traefik is using http to communicate with your app by default. It terminates the tls. To reencrypt the request, you'll need to set traefik.protocol=https. docs.traefik.io/configuration/backends/docker – Br

I already put traefik.default.protocol=https in the container. I have replaced it by traefik.protocol=https. I have the same result : Internal server error. I have looked in traefik web interface and the URL seems good for the backend : https :// 172.21.0.7:443 – Lorenzo

@Lorenzo have you verified the internal server error is coming from traefik and not your application? What logs have you checked? Debugging this will likely need to be a different question. – Br

Yes the application works when I try access it via https with the IP of my dedicated server. – Lorenzo

Some logs, I have found activating debug mode: time="2018-03-22T18:12:24Z" level=debug msg="vulcand/oxy/forward/http: Round trip: https://172.21.0.8:443, code: 500, Length: 21, duration: 9.74602ms tls:version: 303, tls:resume:true, tls:csuite:c02f, tls:server:wiki.domain.tld" – Lorenzo

time="2018-03-22T18:12:30Z" level=debug msg="vulcand/oxy/forward/http: Round trip: http://172.21.0.4:8080, code: 304, Length: 0, duration: 4.449222ms tls:version: 303, tls:resume:true, tls:csuite:c02f, tls:server:wiki.domain.tld" – Lorenzo

Just for help users that didn't have success with this fix in traefik.port: for me, only remove and create the docker network solve the issue. – Pohai

i went to a lot places, but ur simple answer, and simple explanation was the thing for me to get out – Felid

Answer 2

traefik.docker.network must also be the fully qualified network name. Either externally defined, or prefixed with the stack name.

You can alternatively define a default network with providers.docker.network=traefik-network which means you don't have to add the label to every container.

Broughton

Answer 3

Verify Apply:

firewall-cmd --add-masquerade --permanent

FROM: https://www.reddit.com/r/linuxadmin/comments/7iom6e/what_does_firewallcmd_addmasquerade_do/

Masquerading is a fancy term for Source NAT.

firewall-cmd in this instance will be adding an iptables rule, specifically to the POSTROUTING chain in the nat table.

You can see what it has actually done by running iptables -t nat -nvL POSTROUTING. A typical command to manually create a masquerading rule would be iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE, which translates to "For packets leaving interface eth0 after they have been routed, change their source address to the interface address of eth0".

This automatically adds a connection tracking entry so that packets for connections that are masqueraded in this way have their original address and port information reinstated as they return back through the system.

None of this makes your Linux system into a router; that is separate behaviour which is enabled (for IPv4) either by doing sysctl -w net.ipv4.ip_forward=1 or echo 1 > /proc/sys/net/ipv4/ip_forward.

Routing simply means that the system will dumbly traffic it receives according to the destination of that traffic; the iptables NAT stuff allows you to alter the packets which are emitted after that routing takes place.

This is a really simple overview and there is a lot more complexity and possibilities available by configuring it in different ways.

Terbium