python, "urlparse.urlparse(url).hostname" return None value

Question

After loging in on a website I want to collect its links. This I do with this function (using mechanize and urlparse libraries):

br = mechanize.Browser()

.
. #logging in on website
.

for link in br.links():
    url = urlparse.urljoin(link.base_url, link.url)

    hostname = urlparse.urlparse(url).hostname
    path = urlparse.urlparse(url).path

    #print hostname #by printing this I found it to be the source of the None value

    mylinks.append("http://" + hostname + path)

and I get this error message:

    mylinks.append("http://" + hostname + path)
TypeError: cannot concatenate 'str' and 'NoneType' objects

I am not sure on how to fix this, or even if it can be fixed at all. Is there any way to force the function to append even if it would produce a nonworking and weird result for the None value?

Alternatively, what I'm really after in the link is what the link ends with. for example, the html code for one of the links look like this (what I am after is the world "lexik"):

<td class="center">
    <a href="http://UnimportantPartOfLink/lexik>>lexik</a>
</td>

so an alternative route would be if mechanize can just collect this value directly, bypassing the links and None value troubles

Tophet

Can't you just wrap the append statement in a try except block? – Stalactite

Answer 1

Another good way without any try and except block -

Replace hostname = urlparse.urlparse(url).hostname with

hostname = urlparse.urlparse(url).hostname or ''

and similarly path = urlparse.urlparse(url).path with

path = urlparse.urlparse(url).path or ''

Hope this helps !

Radmilla

thanks for the suggestion, that still gave the "TypeError: cannot concatenate 'str' and 'NoneType' objects" message thou – Tophet

that works, neat solution. How does it work, is it like if the first value is None it gives the empty string instead? – Tophet

Yes. If the value of the first value is None, it will be the second one that is assigned. – Ferraro

Answer 2

Why not use a try/except block?

try:
    mylinks.append("http://" + hostname + path)
except TypeError:
    continue

If there's an error, it would just skip the appending and go on with the loop.

Hope this helps!

Ferraro