I/O error(socket error): [Errno 111] Connection refused

Question

I have a program that uses urllib to periodically fetch a url, and I see intermittent errors like :

I/O error(socket error): [Errno 111] Connection refused.

It works 90% of the time, but the othe r10% it fails. If retry the fetch immediately after it fails, it succeeds. I'm unable to figure out why this is so. I tried to see if any ports are available, and they are. Any debugging ideas?

For additional info, the stack trace is:

File "/usr/lib/python2.6/urllib.py", line 203, in open 
    return getattr(self, name)(url)

File "/usr/lib/python2.6/urllib.py", line 342, in open_http
    h.endheaders()

File "/usr/lib/python2.6/httplib.py", line 868, in endheaders
    self._send_output()

File "/usr/lib/python2.6/httplib.py", line 740, in _send_output
    self.send(msg)

File "/usr/lib/python2.6/httplib.py", line 699, in send
    self.connect()

File "/usr/lib/python2.6/httplib.py", line 683, in connect
    self.timeout)

File "/usr/lib/python2.6/socket.py", line 512, in create_connection
    raise error, msg

Edit - A google search isn't very helpful, what I got out of it is that the server I'm fetching from sometimes refuses connections, how can I verify its not a bug in my code and this is indeed the case?

Cruce

What does "I tried to see if any ports are available, and they are." mean? If you are getting an ECONNREFUSED that is an affirmative response - at the raw socket layer - from the remote host saying "I hear you knocking but no one is listening"; this typically results from an overloaded server. Your retry indicates that this is transient as does your 90/10% ratio. How to prove it isn't your code? Telnet to the server port. Write a python socket layer routine to try connections and give you the hit / refused ratio. – Aeschines

Answer 1

Use a packet sniffer like Wireshark to look at what happens. You need to see a SYN-flagged packet outgoing, a SYN+ACK-flagged incoming and then a ACK-flagged outgoing. After that, the port is considered open on the local side.

If you only see the first packet and the error message comes after several seconds of waiting, the other side is not answering at all (like in: unplugged cable, overloaded server, misguided packet was discarded) and your local network stack aborts the connection attempt. If you see RST packets, the host actually denies the connection. If you see "ICMP Port unreachable" or host unreachable packets, a firewall or the target host inform you of the port actually being closed.

Of course you cannot expect the service to be available at all times (consider all the points of failure in between you and the data), so you should try again later.

Troublous

Answer 2

Getting an ECONNREFUSED errno means that your kernel was refused a connection at the other end, so if it's a bug, it's either in your kernel or in the other end. What you can do is to trap the error in a very specific way and try again in a little while, since this seems to work:

# This is Python > 2.5 code
import errno, time

for attempt in range(MAXIMUM_NUMBER_OF_ATTEMPTS):
    try:
        # your urllib call here
    except EnvironmentError as exc: # replace " as " with ", " for Python<2.6
        if exc.errno == errno.ECONNREFUSED:
            time.sleep(A_COUPLE_OF_SECONDS)
        else:
            raise # re-raise otherwise
    else: # we tried, and we had no failure, so
        break
else: # we never broke out of the for loop
    raise RuntimeError("maximum number of unsuccessful attempts reached")

Replace the two all-caps constants with your favourite numbers.

Dietary

Answer 3

I previously had this problem with my EC2 instance (I was serving couchdb to serve resources -- am considering Amazon's S3 for the future).

One thing to check (assuming Ec2) is that the couchdb port is added to your open ports within your security policy.

I specifically encountered

"[Errno 111] Connection refused"

over EC2 when the instance was stopped and started. The problem seems to be a pidfile race. The solution for me was killing couchdb (entirely and properly) via:

pkill -f couchdb

and then restarting with:

/etc/init.d/couchdb restart

Adviser

Answer 4

I'm not exactly sure what's causing this. You can try looking in your socket.py (mine is a different version, so line numbers from the trace don't match, and I'm afraid some other details might not match as well).

Anyway, it seems like a good practice to put your url fetching code in a try: ... except: ... block, and handle this with a short pause and a retry. The URL you're trying to fetch may be down, or too loaded, and that's stuff you'll only be able to handle in with a retry anyway.

Doyledoyley

Answer 5

Its seems that server is not running properly so ensure that with terminal by

telnet ip port

example

telnet localhost 8069

It will return connected to localhost so it indicates that there is no problem with the connection Else it will return Connection refused it indicates that there is problem with the connection

Zwieback