nginx error message - what does "peer" refer to?

Question

In order to debug an nginx error case, I need to fully understand an error log message first. Our nginx writes the particular error log message from time to time.

Log message

"peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream".

What is meant by "peer"?

I would like to know: Does "peer" refer to the upstream, meaning that the upstream closed the connection during ssl handshake, or does it refer to the client, meaning that the client closed connection while the load balancer and the webserver was internally during a handshake?

Setup

• nginx loadbalancer
• 2 webservers (upstreams) running IIS8
• Ssl provider: Comodo

Weismannism

Answer 1

After many hours of debugging we finally found the actual cause of the issue. The error message was produced by a client requesting the nginx without a domain, e.g. https://11.22.33.44/robots.txt. Nginx then forwarded the request to an IIS-server which did not have any default websites bound to https for ip-alone-requests.

The conclusion for the original question is then, that "peer" actually DOES refer to the upstream (the IIS), as the IIS is the one cutting the connection.

The solution we chose to this problem to not get this error in nginx and hereby avoid exposure for clients to send all servers in "down"-mode is to configure the nginx to deny these requests by itself. Another possible solution was to ensure that the IIS behaved nicely for these requests.

Weismannism

Answer 2

Peer refers to upstream in this case. Just because if we take that peer is a client, that would mean that two SSL handshakes (Client -> nginx, nginx -> upstream) happen simultaneously, which doesn't make sense - client have to establish connection and send a query, and only then nginx can choose appropriate upstream to connect to

Vega

A side question: Do you think this error could in any way be provoked by the client by sending specific data or is it an isolated issue between the laod balancer and the webserver? – Aldana

I don't think it's about client – Vega

The error message talks about the peer and upstream separately. It says, "peer closed connection ... while SSL handshaking to upstream" which indicates they are NOT the same. – Lith

Well I can't agree with you. "while SSL handshaking to upstream" is a context of the event, and "peer closed connection in SSL handshake (104: Connection reset by peer)" is the event itself. This error is defined in src/event/ngx_event_openssl.c and it's triggered by SSL connection being closed, and the SSL connection is the upstream in this case – Vega

Answer 3

Your issue might be to do with the order you have concatenated the Comodo .bundle file with your site cert.

You need to place the bundle file after the site cert.

Click this link for more details

EDIT

The peer has to be something on the same "level" as Nginx which, as your issues are related to SSL, has to be OpenSSL.

I would hazard a guess that your OS is Ubuntu 12.x and that OpenSSL is 1.0.1. If so, then the issue is most likely related to an Ubuntu bug.

Seems you either need to upgrade to Ubuntu 13.04 or disable TLS 1.1.

Click this link for more details

Whatever the case, the peer is not the upstream.

Lith

Thank you for pointing at a possible reason for the error message. The certificate is already installed the right way. Seems like its actually the client that "Peer" is refering to. – Aldana

I am still not sure anyway, what peer refers to. – Aldana

Thank you for pointing in your edit, Dayo. I will go dive into this and get back. – Aldana

The server already runs latest ubuntu, so this is not the cause of the problem. We will go try fix the issue by not running SSL between the load balancer and the webservers. – Aldana

I guess that is a viable work around your actual issue but as I said, the peer is not the upstream. You haven't mentioned looking at OpenSSL but it is moot now. Good luck! – Lith

Well I agree with you that the peer could be the client. Would it be possible to provoke the error message byt handshaking with the oad balancer and then cut the wire the moment after the handshake while the load balancer handshakes with the webserver? Could this be what is really going on? Cause this sends the webserver in inoperative mode even though the error is not caused by the webserver. – Aldana

Sorry I was not clear, your issue is between OpenSSL and the upstream Server. The closed connection is usually because the requested cipher method is not supported. – Lith

But we use comodo ssl certificates. Isnt this another ssl vendor than open ssl? Or have I missed some details about the workings of ssl? – Aldana

Another thing: If you say it could be caused by the cifer method, wouldnt the problem occur when handshainkg with client instead of while handshaking with upstream? The client and the load balancer already have done their handshake when the handshake towards the upstream is taking place or am I wrong? – Aldana

I now know that the load balancer runs Ubuntu 14.04 og OpenSSL 1.0.1f 6 Jan 2014 - does the bug still exist in this version? – Aldana

It is possible that the bug still exists: bugs.launchpad.net/ubuntu/+source/openssl/+bug/1305175. On the various follow up queries, I think you might need to open another question to ask those as they are difficult to address in the comments section. – Lith