How to use urllib with username/password authentication in python 3?

Question

Here is my problem with urllib in python 3.

I wrote a piece of code which works well in Python 2.7 and is using urllib2. It goes to the page on Internet (which requires authorization) and grabs me the info from that page.

The real problem for me is that I can't make my code working in python 3.4 because there is no urllib2, and urllib works differently; even after few hours of googling and reading I got nothing. So if somebody can help me to solve this, I'd really appreciate that help.

Here is my code:

    request = urllib2.Request('http://mysite/admin/index.cgi?index=127')
    base64string = base64.encodestring('%s:%s' % ('login', 'password')).replace('\n', '')
    request.add_header("Authorization", "Basic %s" % base64string)
    result = urllib2.urlopen(request)
    resulttext = result.read()

Reggy

Hmmm this may be one of the reasons why most people (including myself) still go for 2.7 ;) As much as it sucks. There may be no other way to overcome this. But good luck. – Ardyce

"I can't make my code working" How so? What exactly is going wrong? – Candidacandidacy

Actually, everything. Since there is no module 'urllib2', the entyre thing doesn't work. I think there must be some way to do the same action with python 3.4, but as I said - after few hours of reading web, I've got nothing. Maybe somebody already had such problem and solved it. – Reggy

Answer 1

Thankfully to you guys I finally figured out the way it works. Here is my code:

request = urllib.request.Request('http://mysite/admin/index.cgi?index=127')
base64string = base64.b64encode(bytes('%s:%s' % ('login', 'password'),'ascii'))
request.add_header("Authorization", "Basic %s" % base64string.decode('utf-8'))
result = urllib.request.urlopen(request)
resulttext = result.read()

After all, there is one more difference with urllib: the resulttext variable in my case had the type of <bytes> instead of <str>, so to do something with text inside it I had to decode it:

text = resulttext.decode(encoding='utf-8',errors='ignore')

Reggy

Answer 2

What about urllib.request ? It seems it has everything you need.

import base64
import urllib.request

request = urllib.request.Request('http://mysite/admin/index.cgi?index=127')
base64string =  bytes('%s:%s' % ('login', 'password'), 'ascii')
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib.request.urlopen(request)
resulttext = result.read()

Bobbiebobbin

Thanks for your response. I've already tried this way. It gets me an error: File "C:\Python34\lib\base64.py", line 522, in _input_type_check raise TypeError(msg) from err TypeError: expected bytes-like object, not str – Reggy

That has nothing to do with urllib, but it's due the new way that Python 3 handles string / bytes string (see for instance python.about.com/od/python30/ss/30_strings_3.htm). I changed the code to a working version. – Bobbiebobbin

Thanks for reply, but still not it. Now: HTTP Error 401: Unauthorized. I'd think that I do something wrong in my way of authorization, but code that I provided with my question works just fine. So there must be something with this urllib.request thing. – Reggy

I thing I found reason for 401 Error with WireShark. With python 3.4 it doesn't encode my login and pass. It just sends "Authorization: Basic b'login:password', when in old python it sent something like Authorization: Basic bW9vb3N6253gFDgbWVhbG9uZQ== Any reason for that? – Reggy

Answer 3

An alternative using OpenerDirector that installs the auth headers for all future urllib requests

login_pass = base64.b64encode(f'{login}:{password}'.encode()).decode()
opener = urllib.request.build_opener()
opener.addheaders = [('Authorization', f'Basic {login_pass}')]
urllib.request.install_opener(opener)

response = urllib.request.urlopen(API_URL)
print(response.read().decode())

A further example using HTTPBasicAuthHandler although a bit more work required if need to send credentials unconditionally:

password_mgr = urllib.request.HTTPPasswordMgrWithPriorAuth()
password_mgr.add_password(None, API_URL, login, password, is_authenticated=True)
auth_handler = request.HTTPBasicAuthHandler(password_mgr)
opener = request.build_opener(auth_handler)
request.install_opener(opener)

response = urllib.request.urlopen(API_URL)
print(response.read().decode())

Margarettamargarette