Permanent gaierror 'Temporary failure in name resolution' after running for a few hours

Question

I have a long running python script, launched with upstart. This script makes quite a lot of requests. Everything works well at first, however after a few hours I start permanently getting the following error for each request:

  File "/opt/a/a-env/local/lib/python2.7/site-packages/atom/client.py", line 119, in request
  File "/opt/a/a-env/local/lib/python2.7/site-packages/atom/http_core.py", line 420, in request
  File "/opt/a/a-env/local/lib/python2.7/site-packages/atom/http_core.py", line 489, in _http_request
  File "/usr/lib/python2.7/httplib.py", line 931, in endheaders
  File "/usr/lib/python2.7/httplib.py", line 794, in _send_output
  File "/usr/lib/python2.7/httplib.py", line 756, in send
  File "/usr/lib/python2.7/httplib.py", line 1134, in connect
  File "/usr/lib/python2.7/socket.py", line 553, in create_connection
gaierror: [Errno -3] Temporary failure in name resolution

This is not a problem with name resolution or DNS, because a simple restart of the application fixes this problem.

I've tried both Python 2.6 a 2.7 and the same situation occurs.

I'm running Linux 2.6.35-30-virtual #61-Ubuntu SMP Tue Oct 11 18:26:36 UTC 2011 x86_64 GNU/Linux

There have been a few posts with this problem, but none with a straightforward explanation or solution:

• https://github.com/ggstuart/greenview/issues/4
• Permanent 'Temporary failure in name resolution' after running for a number of hours

Touchstone

Answer 1

I think this happens when you get in a "too many open files" condition. Next time this happens, try to see how many file descriptors your script has open.

Buroker

Been getting it recently, only 394 file descriptors open across entire system (lsof | wc -l). – Wrigley

Answer 2

The root cause is that /etc/resolv.conf is only read at python startup. To force a refresh of the resolve table, you can run this (on Linux):

import ctypes
libc = ctypes.cdll.LoadLibrary('libc.so.6')
res_init = libc.__res_init
res_init()

Anabas

Please do not post the same answer to multiple questions. If the same information really answers both questions, then one question (usually the newer one) should be closed as a duplicate of the other. You can indicate this by voting to close it as a duplicate or, if you don't have enough reputation for that, raise a flag to indicate that it's a duplicate. Otherwise, be sure you tailor your answer to this question and don't just paste the same answer in multiple places. – Mckean