Web app hangs for several hours in ssl.py at self._sslobj.do_handshake()

Question

I am using Python 2.7.5. I have a web app which queries an API every few minutes and has been working successfully for the last day or so. However, after leaving it sitting for a few hours, I came back to find my program stalled with no activity for several hours. I quit the program and found that it has been stalled in the ssl handshake method for most of the day, during one of the API calls.

Here is the traceback:

...
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 143, in __init__
  self.do_handshake()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 305, in do_handshake
  self._sslobj.do_handshake()

I did a little research, and it seems that this was a problem with the SSL libraries in Python 2.6, but it has since been fixed. I'm wondering why my program got stuck here without throwing an exception or anything.

If there is a way to set a timeout for the SSL handshake method, I would gladly do that, as I would like to avoid my program being stopped indefinitely from something like this again. I am using the Requests HTTP library and this is running on Mac OSX 10.9, if that matters. Suggestions?

EDIT: I did some research, and it seems that others have had this specific problem with SSL, despite the "fix" implemented in 2.6. Not sure what the solution is yet, however. Any help is appreciated.

EDIT 3: Added my solution as an answer to this question.

Stiff

What does openssl s_client -connect mysslservice.example.com:443 return ? It could be waiting for you to provide a client certificate or stuck in a certificate validation that never completes. – Ramonramona

I can't test it right now, but like I said it was working perfectly up until that one call. And when I restarted the service, it started pulling data from the API as it was before. I'm just wondering why it got stuck that one time. – Stiff

I experienced the same error, program was stuck for more than 10 hours until I got home and restarted it. I`ll use your workaround. Python 2.7.3 in Ubuntu 13.10. – Obeded

2.7.5+ I mean. I was looking at a remote machine. – Obeded

Sounds good. Yeah that solution should definitely take care of this issue, at least in any situation I can imagine. I've added it as an official "answer" so if it helps you, I would appreciate it if you toss me an upvote :) – Stiff

The link in the first edit is broken, but here is a working one to the same content: mail.python.org/pipermail/python-list/2012-June/638443.html – Neuroglia

Answer 1

After browsing through the Python section of Stack Overflow, I found something which may not fix the core issue causing the problem, but is definitely good enough to handle any situation in which this issue pops up. The following question has various solutions which will throw some sort of exception if a function takes too long to complete. That is how I solved this issue in the end. The top answer is UNIX-only, however there are some others which use threading and work on every platform:

Timeout function if it takes too long to finish

This is a strange issue which is honestly pretty hard to reproduce. I've only seen it twice after thousands and thousands of API calls. I think it's pretty unlikely that someone will arrive at a better solution than this one, which is sort of a hack, but definitely solves the problem. You can throw an exception, then either attempt the SSL connection again, or continue with another part of your program.

I think my answer will suffice for now, but if someone has something better, feel free to offer it forward. Honestly, it seems like the only fix for the underlying issue might be a bug fix in the actual ssl.py library, but it's impossible for me to say for sure.

Stiff

Were you able to find a concrete solution to this? And more importantly, what is triggering this "problem"? – Guff

I seem to be getting it about every 10 minutes using threading in Python 3 – Muntjac

Answer 2

I ran into this and was able to work around it using socket.setdefaulttimeout. socket.setdefaulttimeout modifies the behavior of all sockets created after calling it, which is convenient if the socket is encapsulated in library code. If you have access to the specific SSL socket you can probably use socket.settimeout.

Maryn

Answer 3

In my case, the connection stuck in do_handshake(), I add a sleep before handshake, it seems solved.

def patch_ssl():
    import ssl
    import time
    def new_wrap_socket(self, sock, server_side=False,
                        do_handshake_on_connect=True,
                        suppress_ragged_eofs=True,
                        server_hostname=None, session=None):
        # SSLSocket class handles server_hostname encoding before it calls
        # ctx._wrap_socket()
        time.sleep(0.005)
        return self.sslsocket_class._create(
            sock=sock,
            server_side=server_side,
            do_handshake_on_connect=do_handshake_on_connect,
            suppress_ragged_eofs=suppress_ragged_eofs,
            server_hostname=server_hostname,
            context=self,
            session=session
        )
    ssl.SSLContext.old_wrap_sock = ssl.SSLContext.wrap_socket
    ssl.SSLContext.wrap_socket = new_wrap_socket
patch_ssl()

Tay