Dynamic proxy_pass in nginx to another pod in Kubernetes

Question

I'm trying to create an nginx proxy that forwards requests to /<service> to http://<service>. I first tried the following:

location ~ ^/(.+)$ {
    set $backend "http://$1:80";
    proxy_pass $backend;
}

But it fails saying something like (when calling /myservice):

[error] 7741#0: *1 no resolver defined to resolve http://myservice

Since myservice is not externally accessible I've tried to install go-dnsmasq as a sidecar in the same pod and I try to use it for DNS resolution (like I've seen in this example) and change my nginx config to look like this:

location ~ ^/(.+)$ {
        resolver 127.0.0.1:53;
        set $backend "http://$1:80";
        proxy_pass $backend;
}

But now nginx fails with:

[error] 9#9: *734 myservice could not be resolved (2: Server failure), client: 127.0.0.1, server: nginx-proxy, request: "GET /myservice HTTP/1.1", host: "localhost:8080"
127.0.0.1 - xxx [30/May/2016:10:34:23 +0000] "GET /myservice HTTP/1.1" 502 173 "-" "curl/7.38.0" "-"

My Kubernetes pod looks like this:

spec:
  containers:
    - name: nginx
      image: "nginx:1.10.0"
      ports:
        - containerPort: 8080
          name: "external"
          protocol: "TCP"
    - name: dnsmasq
      image: "janeczku/go-dnsmasq:release-1.0.5"
      args:
        - --listen
        - "0.0.0.0:53"

Running netstat -ntlp in the dnsmasq container gives me:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      -
tcp        0      0 :::53                   :::*                    LISTEN      1/go-dnsmasq

And running nmap --min-parallelism 100 -sT -sU localhost in the nginx container:

Starting Nmap 6.47 ( http://nmap.org ) at 2016-05-30 10:33 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00055s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 1997 closed ports
PORT     STATE SERVICE
53/tcp   open  domain
8080/tcp open  http-proxy
53/udp   open  domain

So it seems that dnsmasq and nginx are indeed up and running? What could I be doing wrong?

Jabalpur

Answer 1

After much research and trial and error I managed to solve this. First I changed the pod specification to:

spec:
  containers:
    - name: nginx
      image: "nginx:1.10.0"
      ports:
        - containerPort: 8080
          name: "external"
          protocol: "TCP"
    - name: dnsmasq
      image: "janeczku/go-dnsmasq:release-1.0.5"
      args:
        - --listen
        - "127.0.0.1:53"
        - --default-resolver
        - --append-search-domains
        - --hostsfile=/etc/hosts
        - --verbose

then I also had to disable the ipv6 for the resolver in nginx:

location ~ ^/(.+)$ {
        resolver 127.0.0.1:53 ipv6=off;
        set $backend "http://$1:80";
        proxy_pass $backend;
}

Then it works as expected!

Jabalpur

This is great. Thank you. Not sure why kube-dns as a resolver doesn't work. – Luxuriance

You just literally saved my life. Had no idea on how to run another container as a sidecar to my deployments. THANKS! – Rolle

For me existing static proxy_passes failed after adding the sidecar, but removing the "--default-resolver" flag fixed the issue 🔥 – Rie

Answer 2

I resolved this by coredns docker ： my nginx and coredns are all deploy on host

step1: config Corefile in Corefile maybe you should change k8s master config refer: https://coredns.io/plugins/kubernetes/

sudo mkdir /etc/coredns; sudo tee /etc/coredns/Corefile   <<-'EOF' .:53 {
    log
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       endpoint http://172.31.88.71:8080
       pods insecure
       upstream
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance } EOF

step2:config docker and then start it

tee coreos.sh   <<-'EOF'
 docker run --restart=always  -idt --name coredns \
 -v /etc/coredns/Corefile:/etc/coredns/Corefile \
 -v /home/ec2-user/.kube/config:/etc/coredns/kubeconfig \
 -p 53:53/udp \
 coredns/coredns:1.6.9 \
 -conf /etc/coredns/Corefile
EOF

step3: config nginx and then reload

resolver 127.0.0.1 valid=60s ipv6=off;

Pembroke