nginx does not automatically pick up dns changes in swarm

Question

I'm running nginx via lets-nginx in the default nginx configuration (as per the lets-nginx project) in a docker swarm:

services:

  ssl:
    image: smashwilson/lets-nginx
    networks:
      - backend
    environment:
      - [email protected]
      - DOMAIN=api.finestructure.co
      - UPSTREAM=api:5000
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - letsencrypt:/etc/letsencrypt
      - dhparam_cache:/cache

  api:
    image: registry.gitlab.com/project_name/image_name:0.1
    networks:
      - backend
    environment:
      - APP_SETTINGS=/api.cfg
    configs:
      - source: api_config
        target: /api.cfg
    command:
      - run
      - -w
      - tornado
      - -p
      - "5000"

api is a flask app that runs on port 5000 on the swarm overlay network backend.

When services are initially started up everything works fine. However, whenever I update the api in a way that makes the api container move between nodes in the three node swarm, nginx fails to route traffic to the new container.

I can see in the nginx logs that it sticks to the old internal ip, for instance 10.0.0.2, when the new container is now on 10.0.0.4.

In order to make nginx 'see' the new IP I need to either restart the nginx container or docker exec into it and kill -HUP the nginx process.

Is there a better and automatic way to make the nginx container refresh its name resolution?

Stowe

You can try setting proxy_pass to a variable and define a resolver to force re-resolution, like here: serverfault.com/questions/240476/… – Nonagenarian

Good tip - but I've double checked and the default config includes proxy_pass http://${UPSTREAM}; which is substituted with the configured hostname by lets-nginx – Stowe

Or is it relevant that this setting comes with a set ... clause? I'll give that a try. – Stowe

Yes its relevant to set a variable with set and also to define a resolver – Nonagenarian

Ok, I've tried this with the suggested config changes (resolver and set ...) but it seems it still doesn't pick up changes in service ips. – Stowe

@sas, That ${UPSTREAM} is a environment variable and not a nginx variable. Without a nginx variable this won't work. – Wise

Yes, I know - this is replaced by an outside substitution via the launcher script in lets-nginx – Stowe

Ok, apparently adding valid=10s to override TTL does the trick, at least here in my case. – Stowe

Answer 1

Thanks to @Moema's pointer I've come up with a solution to this. The default configuration of lets-nginx needs to be tweaked as follows to make nginx pick up IP changes:

  resolver 127.0.0.11 ipv6=off valid=10s;
  set $upstream http://${UPSTREAM};
  proxy_pass $upstream;

This uses docker swarm's resolver with a TTL and sets a variable, forcing nginx to refresh name lookups in the swarm.

Stowe

Answer 2

Remember that when you use set you need to generate the entire URL by yourself.

I was using nginx in a compose to proxy a zuul gateway :

        location /api/v1/ {
           proxy_set_header X-Forwarded-Host $host:$server_port;
           proxy_pass http://rs-gateway:9030/api/v1/;
        }

        location /zuul/api/v1/ {
           proxy_set_header X-Forwarded-Host $host:$server_port;
           proxy_pass http://rs-gateway:9030/zuul/api/v1/;
        }

Now with Swarm it looks like that :

        location ~ ^(/zuul)?/api/v1/(.*)$ {
            set $upstream http://rs-gateway:9030$1/api/v1/$2$is_args$args;
            proxy_pass $upstream;
            # Set headers
            proxy_set_header X-Forwarded-Host $host:$server_port;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $http_connection;
        }

Regex are good but don't forget to insert GET params into the generated URL by yourself.

Snack