Nginx will not start with host not found in upstream

Question

I use nginx to proxy and hold persistent connections to far away servers for me.

I have configured about 15 blocks similar to this example:

upstream rinu-test {
    server test.rinu.test:443;
    keepalive 20;
}
server {
    listen 80;
    server_name test.rinu.test;
    location / {
        proxy_pass https://rinu-test;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $http_host;
    }
}

The problem is, if the hostname cannot be resolved in one or more of the upstream blocks, nginx will not (re)start. I can't use static IPs either, some of these hosts explicitly said not to do that because IPs will change. Every other solution I've seen to this error message says to get rid of upstream and do everything in the location block. That is not possible here because keepalive is only available under upstream.

I can temporarily afford to lose one server but not all 15.

Edit: Turns out nginx is not suitable for this use case. An alternative backend (upstream) keepalive proxy should be used. A custom Node.js alternative is in my answer. So far, I haven't found any other alternatives that actually work.

Species

There are two things you can try. Change proxy_pass https://rinu-test; to proxy_pass $proxyurl; and before that you can set the variable set $proxyurl $scheme://$host$request_uri And next is to try use variable in upstream, I have not tested the 2nd option and can't verify yet. But using a variable in proxy_pass disables dns caching in nginx – Shantay

Proxying without the upstream is pointless. Variables cannot be used in upstream. – Species

I meant you could try something like proxy_pass https://rinu-test$request_uri; – Shantay

How about exploring HAProxy instead of nginx for this? If you use upstreams then this use case cannot be handled by default – Shantay

I tried HAProxy but it didn't work. It did proxy but didn't keep connections open or failed to reuse them. – Species

It should have keep-alive enabled by default? #46967313 – Shantay

That is the theory yes but I guess it only works on the frontend side. My testing has clearly shown that it doesn't work on backend side. I tried all the options, spent like 4 hours on it. – Species

Answer 1

Earlier versions of nginx (before 1.1.4), which already powered a huge number of the most visited websites worldwide (and some still do even nowadays, if the server headers are to be believed), didn't even support keepalive on the upstream side, because there is very little benefit for doing so in the datacentre setting, unless you have a non-trivial latency between your various hosts; see https://serverfault.com/a/883019/110020 for some explanation.

Basically, unless you know you specifically need keepalive between your upstream and front-end, chances are it's only making your architecture less resilient and worse-off.

(Note that your current solution is also wrong because a change in the IP address will likewise go undetected, because you're doing hostname resolution at config reload only; so, even if nginx does start, it'll basically stop working once IP addresses of the upstream servers do change.)

Potential solutions, pick one:

• The best solution would seem to just get rid of upstream keepalive as likely unnecessary in a datacentre environment, and use variables with proxy_pass for up-to-date DNS resolution for each request (nginx is still smart enough to still do the caching of such resolutions)

• Another option would be to get a paid version of nginx through a commercial subscription, which has a resolve parameter for the server directive within the upstream context.

• Finally, another thing to try might be to use a set variable and/or a map to specify the servers within upstream; this is neither confirmed nor denied to have been implemented; e.g., it may or may not work.

Conventionalism

I know what I'm doing. Keepalive has made requests to external service providers up to a whole 1 second faster and saving time in that application is critical. – Species

Variables cannot be used to define the server in upstream. – Species

Paid nginx is too expensive. It's cheaper to rewrite the application that needs this proxy. Which is already in progress but I would estimate another 3-4 years to complete it. – Species

@Species TBH, if keepalive between the front-end and backend shaves a whole second, it sounds like the whole architecture of the application may be quite suboptimal. Why is there so much latency? Do you proxy_pass between Europe and America/Asia? If so, might as well make more sense to put some extra front-end servers closer to where your backends actually live. Else, there shouldn't be that much latency, or, perhaps if you're doing in-the-field IoT, then the data being proxied should probably be processed outside of the context of a proxy server first. – Conventionalism

There are about 100 independent service providers all over the world, mainly in America and Europe. Several are queried in parallel, at most I've seen about 20. In one service provider's example average response times dropped from 1.3 to 0.5 seconds. The lowest gain was 150 ms. So far none have been slower with the keepalive proxy. The application is in PHP which can make requests in parallel but due to it's short lifespan nature can't hold connections open. – Species

@rinu, I don't see how you could have 1.3s savings from a single reuse of a connection through keepalive, it seems like you may be measuring something else than the keepalive savings. Also, I'm kinda confused about the usecase here — you're basically trying to query 100 independent providers through PHP, and use nginx as a connection cache? Then your question is very poorly formulated for what you're trying to accomplish, as there'd be many other better ways to do keepalive, possibly even with nginx stream — you unfairly restrict the scope of the solution through an incomplete specification. – Conventionalism

That request on average used to take 1.3s and now take 0.5s. Keealive saved the difference, 0.8s. Request times are measured by curl. At this point I'm convinced I shouldn't even use nginx for this, alternatives are being investigated. – Species

Answer 2

Your scenario is very similar to the one when using aws ELB as uptreams in where is critical to resolve the proper IP of the defined domain.

The first thing you need to do and ensure is that the DNS servers you are using can resolve to your domains, then you could create your config like this:

resolver 10.0.0.2 valid=300s;
resolver_timeout 10s;

location /foo {
    set $foo_backend_servers foo_backends.example.com;
    proxy_pass http://$foo_backend_servers;
 }

location /bar {
    set $bar_backend_servers bar_backends.example.com;
    proxy_pass http://$bar_backend_servers;
 }

Notice the resolver 10.0.0.2 it should be IP of the DNS server that works and answer your queries, depending on your setup this could be a local cache service like unbound. and then just use resolve 127.0.0.1

Now, is very important to use a variable to specify the domain name, from the docs:

When you use a variable to specify the domain name in the proxy_pass directive, NGINX re‑resolves the domain name when its TTL expires.

You could check your resolver by using tools like dig for example:

$ dig +short stackoverflow.com

In case is a must to use keepalive in the upstreams, and if is not an option to use Nginx +, then you could give a try to openresty balancer, you will need to use/implement lua-resty-dns

Gamesmanship

Could you please go into more detail about openresty, lua and how these could be used here? I haven't heard of these before and it's not obvious from the documentation how to even install these or how to apply to my use case. – Species

Hi @rinu, I found this gist probably can give you some more ideas: gist.github.com/toritori0318/f9be21fb8df9e4d5768bb5f484567175 lua + nginx works pretty nice also you could extend and implement a WAF etc, and cover many of the things missing on the standard Nginx – Gamesmanship

Answer 3

A one possible solution is to involve a local DNS cache. It can be a local DNS server like Bind or Dnsmasq (with some crafty configuration, note that nginx can also use specified dns server in place of the system default), or just maintaining the cache in hosts file.

It seems that using hosts file with some scripting is quite straightforward way. The hosts file should be spitted into the static and dynamic parts (i.e. cat hosts.static hosts.dynamic > hosts), and the dynamic part should be generated (and updated) automatically by a script.

Perhaps it make sense to check from time to time the hostnames for changing IPs, and update hosts file and reload configuration in nginx on changes. In case of some hostname cannot be resolved the old IP or some default IP (like 127.0.1.9) should be used.

If you don't need the hostnames in the nginx config file (i.e., IPs are enough), the upstream section with IPs (resolved hostnames) can be generated by a script and included into nginx config — and no need to touch the hosts file in such case.

Hybridism

We already had the DNS cache idea and it will be investigated soon. My sysadmin had a big laugh about the generated hosts file. Guess it's not happening. But I like your answer anyway, so far it's the only one that would actually work. – Species

@rinu, could you please mention some constructive argument what is wrong with using hosts file? – Hybridism

Nothing wrong with it which is why I voted you up. Managing this in all the servers is just too much work. Alternatives to nginx or a better general DNS service are way easier to manage. – Species

Answer 4

I put the resolve parameter on server and you need to set the Nginx Resolver in nginx.conf as below:

/etc/nginx/nginx.conf:

http {
    resolver 192.168.0.2 ipv6=off valid=40s;  # The DNS IP server
} 

Site.conf:

upstream rinu-test {
    server test.rinu.test:443;
    keepalive 20;
}

Solleret

i think that is for Nginx+, but if you set location /foo { set $foo_backend_servers foo_backends.example.com; proxy_pass http://$foo_backend_servers; } will work. If not solve post your conf and logs – Solleret

nginx.org/r/resolver is a standard directive, it's not restricted to nginx plus at all; it's the resolve keyword within the upstream context that's Plus-only; however, this question completely ignores the keepalive issue that the OP specifically mentions that they require. – Conventionalism

you are correct, resolver is open and the resolve parameter form the server command is a on-the-fly reconfiguration option. If you use the variable you will lost keepalive and if you use the upstream you will lost a DNS check that will check only in startup time. – Solleret

This doe snot seem to wokr to solve the original problem - setting the global resolver option does nothing: nginx still doesn't start/restart it if tcan't resolve hosts in the upstream block. – Overslaugh

The name always must be resolved, but the resolved offer the option to keep the DNS up-to-date without lost the upstream configuration. Many users try to use the dns directly in the proxy_pass and fail when whatever lag impact the DNS resolution, besides remove optional confirations in the upstream like keepalive and different LB algorithm – Solleret

Answer 5

My problem was container related. I'm using docker compose to create the nginx container, plus the app container. When setting network_mode: host in the app container config in docker-compose.yml, nginx was unable to find the upstream app container. Removing this fixed the problem.

Dornick

Answer 6

we can resolve it temporarily

cd /etc
sudo vim resolv.conf
i
nameserver 8.8.8.8 
:wq

then do sudo nginx -t restart nginx it will work for the momment

Annal

Answer 7

An alternative is to write a new service that only does what I want. The following replaces nginx for proxying https connections using Node.js

const http = require('http');
const https = require('https');

const httpsKeepAliveAgent = new https.Agent({ keepAlive: true });

http.createServer(onRequest).listen(3000);

function onRequest(client_req, client_res) {
    https.pipe(
        protocol.request({
            host: client_req.headers.host,
            port: 443,
            path: client_req.url,
            method: client_req.method,
            headers: client_req.headers,
            agent: httpsKeepAliveAgent
        }, (res) => {
            res.pipe(client_res);
        }).on('error', (e) => {
            client_res.end();
        })
    );
}

Example usage: curl http://localhost:3000/request_uri -H "Host: test.rinu.test" which is equivalent to: curl https://test.rinu.test/request_uri

Species

This solution is just out of the question scope, since the question is about nginx only (i.e. how to solve the issue with nginx). It is need to reformulate the question to make it conform to this solution :) – Hybridism

In it's current form the question doesn't have an answer. What I'm asking is simply not possible. I also agree that this answer is no exception. But in the end this is how I solved my problem which is why I included it here. – Species