urllib.urlretrieve with custom header

Question

I am trying to retrieve a file using urlretrieve, while adding a custom header.

While checking the codesource of urllib.request I realized urlopen can take a Request object in parameter instead of just a string, allowing to put the header I want. But if I try to do the same with urlretrieve, I get a TypeError: expected string or bytes-like object as mentionned in this other post.

What I ended up doing is rewriting my own urlretrieve, removing the line throwing the error (that line is irrelevant in my use case).

It works fine but I am wondering if there is a better/cleaner way of doing it, rather than rewriting my own urlretrieve. If it is possible to pass a custom header to urlopen, it feels like it should be possible to do the same with urlretrieve?

Kinin

Try to use urllib2. i got also same error one time.but after changing urllib2 it will works – Caldron

Answer 1

I found a way where you only have to add a few extra lines of code...

import urllib.request

opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)
urllib.request.urlretrieve("type URL here", "path/file_name")

Should you wish to learn about the details you can refer to the python documentation: https://docs.python.org/3/library/urllib.request.html

Seventeen

Answer 2

The urllib.request.urlretrieve() use inside urllib.request.urlopen() (at least in Python 3). So you can use same way how you can influence behavior of urlopen.

When urlopen(params) is invoked it actually first looks at the special global variable urllib.request._opener and if it is None then the urlopen set the variable with default set of openers otherwise it will keep it as it. In the next step it will call urllib.request._opener.open(<urlopen_params>) (in next sections I will refer urllib.request._opener only as opener).

The opener.open() contains list of handlers for different protocols. When the opener.open() is called then it will do this actions:

1. Creates from URL urllib.request.Request object (or if you provide directly the Request it will just use it).
2. From the Request object is extracted the protocol (it deduced from URL scheme).
3. Based on the protocol it will try lookup and use those methods:
   • protocol_request (e.g. http_request) - it used for pre-process the request before the connection is opened.
   • protocol_open - actually creates connection with the remote server
   • protocol_response - process the response from the server
   • for other methods look at the Python's documentation

For your own opener you have to do those 3 steps:

1. Create own handler
2. Build list of handlers contains your custom handler (function urllib.request.build_opener)
3. Install the new opener into urllib.request._opener (function urllib.request.install_opener)

The urllib.request.build_opener creates opener which contains your custom handler and add default openers except handlers from which is your custom handler inherited.

So for adding custom header you can write something like this:

import urllib.request as req

class MyHTTP(req.HTTPHandler):
    def http_request(self, req):
        req.headers["MyHeader"] = "Content of my header"
        return super().http_request(req)

opener = req.build_opener(MyHTTP())
req.install_opener(opener)

From this point when you call urllib.request.urlretrieve() or anything which is using the urlopen() it will use for HTTP communication your handler. When you want to get back to default handlers you can just call:

import urllib.request as req   

req.install_opener(req.build_opener())

To be honest I don't know if it is better/cleaner solution then yours but it uses prepared mechanisms in the urllib.

Pyrrha

I feel this is really overkill but I award you the bounty anyway as it was really interesting answer concerning urllib. Anyway I switched to using the requests lib instead – Kinin