Socket error: Address already in use

Question

I have a CherryPy script that I frequently run to start a server. Today I was having to start and stop it a few times to fix some bugs in a config file, and I guess the socket didn't close all the way because when I tried to start it up again I got this issue:

[23/Mar/2015:14:08:00] ENGINE Listening for SIGHUP.
[23/Mar/2015:14:08:00] ENGINE Listening for SIGTERM.
[23/Mar/2015:14:08:00] ENGINE Listening for SIGUSR1.
[23/Mar/2015:14:08:00] ENGINE Bus STARTING
CherryPy Checker:
The Application mounted at '' has an empty config.

[23/Mar/2015:14:08:00] ENGINE Started monitor thread 'Autoreloader'.
[23/Mar/2015:14:08:00] ENGINE Started monitor thread '_TimeoutMonitor'.
[23/Mar/2015:14:08:00] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
  File "/home/andrew/virtualenvs/mikernels/lib/python2.7/site-packages/cherrypy/process/servers.py", line 188, in _start_http_thread
    self.httpserver.start()
  File "/home/andrew/virtualenvs/mikernels/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1848, in start
    raise socket.error(msg)
error: No socket could be created

I edited CherryPy's wsgiserver2.py to see the details of the socket.error and error.strerror was

98  (98, 'Address already in use') Address already in use

Meanwhile my socket is constructed as:

af = 2
socktype = 1
proto = 6
canonname = ''
sa = ('0.0.0.0', 2112)
self.bind(af, socktype, proto)

(that's not exact code but that's what the values are when the error is fired)

I checked netstat and didn't see anything listening on port 2112, what could be causing the problem and how can I go about diagnosing it?

Thanks!

Tigerish

Answer 1

You can try the following

from socket import *

sock=socket()
sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
# then bind

From the docs:

The SO_REUSEADDR flag tells the kernel to reuse a local socket in TIME_WAIT state, without waiting for its natural timeout to expire.

Here's the complete explanation:

Running an example several times with too small delay between executions, could lead to this error:

socket.error: [Errno 98] Address already in use

This is because the previous execution has left the socket in a TIME_WAIT state, and can’t be immediately reused.

There is a socket flag to set, in order to prevent this, socket.SO_REUSEADDR:

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))

Tithe

Thanks, but I see in the code I'm running that CherryPy actually does this. Line 1884: self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) I put a print statement after that line to see if it wasn't being executed but it seems like it is. – Tigerish

The problem seems to be on self.socket.bind(bind_addr) with bind_addr ('0.0.0.0', 2112) – Tigerish

@AndrewLatham, is self.socket the socket that is used for binding? – Tithe

Yes. Also I tried netstat -n -A inet and saw a lot of TIME_WAIT, CLOSE_WAIT, and ESTABLISHED stuff on 2112. – Tigerish

@AndrewLatham, then it's not a Python's problem. You can use ps ax | grep 2112 to try to find programs using this port. After that, you can try to kill all python's processes: killall python. – Tithe

OK, thanks. There was a telnet listening, I killed it, things still aren't starting, maybe I need to wait a few minutes... although netstat -n A inet still has tons of stuff on 2112 even thouse ps ax | grep 2112 comes up with nothing (except the grep itself) – Tigerish

@AndrewLatham, well if you want this socket to be free right now, reboot the device you're running the script on. You can also wait for some time, but this can be too long. – Tithe

Ends up they were all caused by the same process that just failed to die every time I started it. ps -ax -U andrew | grep server.py and killing all the related pids did that. Thanks for your help though! – Tigerish

Thank you so much. This fixed the problem entirely for me. I'm glad my test suite caught this. – Choong

Not work for cli = socket.socket(socket.AF_UNIX, socket.SO_REUSEADDR, 1) – Duplet

@Mithril, the second argument of socket.socket is the socket's type, one of the SOCK_ constants. You can't set socket's options at initialization, you have to use setsockopt as stated in my answer. – Tithe

@Tithe Is it possible to clear the address immediately after use i.e. to put the socket not in a TIME_WAIT state after finishing listening? Some kind of close() method that "frees" the address? I tried socket.close() which doesn't work – Inandin

Answer 2

You could find the process and kill it by doing:

ps aux | grep python

, finding the process ID, and stopping it manually by doing:

sudo kill -9 PID

replacing PID with your PID.

I often have to do this while testing with Flask/CherryPy. Would be interested to see if there's an easier way (for e.g. to prevent it in the first place)

Goldsmith

Wouldn't it be easier to killall python or even sudo killall python? – Tithe

what does -9 mean? – Ashwell

@Ashwell Instead of saying "close please" it just... stops running that process' code. This always translates into data loss; it's just a matter of whether you needed that data or not. – Humidify

@Tithe you mean killall -s 9 python – Gyral

Answer 3

Much more easier to do it by:

Check the PID(:5000 is the host since I've been running on 127.0.0.1:5000):
$ lsof -i :5000
Then kill it:
$ sudo kill -9 PID

Forrer