How to close urllib2 connection? [duplicate]

Question

I have made a program using urllib2 that makes a lot of connections across the web. I noticed that eventually that this can be DDoS worthy; I would like to know how to close down each connection after I have done my business to prevent such an attack.

The code I am using to open a connection is:

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
r = opener.open("http://www.python.org)
html = r.read()

Chesney

Look at this related question: #1523136 but pay attention to 2nd and 3rd answers which are more accurate. – Turk

Answer 1

I assume you are opening them with the urlopen() function. Its documentation states:

This function returns a file-like object with two additional methods:

As a file-like object, it will have a close method which you can call:

connection = urllib2.urlopen(url)
# Do cool stuff in here.
connection.close()

Update: Using the code you added to your question:

>>> import urllib2
>>> import cookielib
>>> cj = cookielib.CookieJar()
>>> opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
>>> r = opener.open("http://www.python.org")
>>> html = r.read()
>>> r.close??
Type:  instancemethod
Base Class: <type 'instancemethod'>
String Form: <bound method addinfourl.close of <addinfourl at 150857644 whose fp = <socket._fileobject object at 0x8fd48ec>>>
Namespace: Interactive
File:  /usr/lib/python2.6/urllib.py
Definition: r.close(self)
Source:
    def close(self):
        self.read = None
        self.readline = None
        self.readlines = None
        self.fileno = None
        if self.fp: self.fp.close()
        self.fp = None

So the close() method exists and actually does something:

>>> r.close()
>>> r.read()
------------------------------------------------------------
Traceback (most recent call last):
  File "<ipython console>", line 1, in <module>
TypeError: 'NoneType' object is not callable

Lake

@Chesney - not on my machine. I updated my answer to use the code you added to your question. – Lake

Answer 2

your question is extremely vague.

but here is an example of closing a connection after use:

f = urllib2.urlopen(req)
f.read()
f.close()

Polarity

Close is not defined; there has to be some other proper way. – Chesney

close() is defined on the urlopen return (file-like object) – Polarity

the close is just def close: pass – Turk

@Senthil Kumaran - not on my machine (Ubuntu 10.10/Python 2.6.6) it isn't. The comment formatting will make this a bit hard to read, but on my machine it does self.read = None; self.readline = None; self.readlines = None; self.fileno = None; if self.fp: self.fp.close(); self.fp = None i.e., it closes the underlying file object and overwrites the read methods with None so they cannot be used. – Lake

Answer 3

The socket connection closes automatically, once the response is obtained. So you don't explicitly close the urllopen object, it happens automatically at the socket level.

Turk

No it doesn't. Set up a connection: conn = urllib2.urlopen('http://www.google.com/'), and then check the status of the socket: print conn.fp.closed which will print False as you'd hope. Read the data: devnull = conn.read(), and now check the status of the socket: print conn.fp.closed which still prints False, i.e., the socket is still open. – Lake

Admittedly it may be different on your operating system / Python version, but this just goes to show you cannot rely on this happening implicitly. Explicit is better than implicit... – Lake

The connection actually is closed whenever you read() the response, you can verify this by looking at the read() method of HTTPResponse, using netstat, etc. – Marrano