Python urllib urlopen not working

A

8

16

I am just trying to fetch data from a live web by using the urllib module, so I wrote a simple example

Here is my code:

import urllib

sock = urllib.request.urlopen("http://diveintopython.org/") 
htmlSource = sock.read()                            
sock.close()                                        
print (htmlSource)

But I got error like:

Traceback (most recent call last):
  File "D:\test.py", line 3, in <module>
    sock = urllib.request.urlopen("http://diveintopython.org/") 
AttributeError: 'module' object has no attribute 'request'

Angloirish answered 16/9, 2014 at 7:36 Comment(0)

Z

26

You are reading the wrong documentation or the wrong Python interpreter version. You tried to use the Python 3 library in Python 2.

Use:

import urllib2

sock = urllib2.urlopen("http://diveintopython.org/") 
htmlSource = sock.read()                            
sock.close()                                        
print htmlSource

The Python 2 urllib2 library was replaced by urllib.request in Python 3.

Zacatecas answered 16/9, 2014 at 7:38 Comment(0)

E

7

import requests
import urllib

link = "http://www.somesite.com/details.pl?urn=2344"

f = urllib.request.urlopen(link)
myfile = f.read()

writeFileObj = open('output.xml', 'wb')
writeFileObj.write(myfile)
writeFileObj.close()

Erratic answered 23/4, 2017 at 19:44 Comment(2)

Usually a good answer doesn't contain just code but a little information on what exactly got fixed. – Postdiluvian 23/4, 2017 at 20:34

isn't requests an different module that you're not actually using in this answer? – Enchanting 1/7, 2022 at 21:10

C

6

In Python3 you can use urllib or urllib3

urllib:

import urllib.request
with urllib.request.urlopen('http://docs.python.org') as response:
    htmlSource = response.read()

urllib3:

import urllib3
http = urllib3.PoolManager()
r = http.request('GET', 'http://docs.python.org')
htmlSource = r.data

More details could be found in the urllib or python documentation.

Cogitative answered 6/3, 2018 at 15:59 Comment(0)

A

3

This is what i use to get data from urls, its nice because you could save the file at the same time if you need it:

import urllib

result = urllib.urlretrieve("http://diveintopython.org/")

print open(result[0]).read()

output:

'<!DOCTYPE html><body style="padding:0; margin:0;"><iframe src="http://mcc.godaddy.com/park/pKMcpaMuM2WwoTq1LzRhLzI0" style="visibility: visible;height: 2000px;" allowtransparency="true" marginheight="0" marginwidth="0" frameborder="0" scrolling="no" width="100%"></iframe></body></html>'

Edit: urlretrieve works in python 2 and 3

Anyplace answered 16/9, 2014 at 7:55 Comment(4)

urlretrieve is available on Python 3 as well as part of the legacy interface. If all you wanted was the data in memory, then urlretrieve is the wrong tool; you downloaded the data to disk then opened it again from the file. – Zacatecas 16/9, 2014 at 8:34

Moreover, this doesn't even attempt to answer the question stated. – Zacatecas 16/9, 2014 at 8:35

well i stated that urlretrieve is nice because you can save the file ... thats the whole point. if someone tells me he wants to fetch data from the internet why not suggesting something like that. – Anyplace 16/9, 2014 at 8:48

and i didnt know that it works in python 3 ;-) though it is outdated or something – Anyplace 16/9, 2014 at 8:49

A

0

Make sure you import requests from urllib, then try this format, it worked for me:

from urllib import request
urllib.request.urlopen( )

Amaryllis answered 18/7, 2018 at 9:27 Comment(0)

D

0

I just queried the same question which is now over 5 years old.

Please note that the URL given is also old, so i substituted the python welcome page.

We can use the requests module in python 3.

I use python 3 and the solution is below:

import requests

r = requests.get('https://www.python.org/')
t = r.text

print(t)

This works and is clean.

Distress answered 9/2, 2020 at 9:9 Comment(0)

O

0

For python 3 the correct way should be:

import cv2
import numpy as np
import urllib.request

req = urllib.request.urlopen('http://answers.opencv.org/upfiles/logo_2.png')
arr = np.asarray(bytearray(req.read()), dtype=np.uint8)
img = cv2.imdecode(arr, -1) # 'Load it as it is'

cv2.imshow('image_name', img)
if cv2.waitKey() & 0xff == 27: quit()

Here you can find the documentation related to urllib.request

Ogee answered 31/3, 2020 at 17:2 Comment(0)

S

-1

Use this

    import cv2
    import  numpy as np
    import urllib //import urllib using pip
    import requests // import requests using pip`enter code here`
    url = "write your url"
    while True:
    imgresp = urllib.request.urlopen(url)
    imgnp = np.array(bytearray(imgresp.read()),dtype=np.uint8)
    img = cv2.imdecode(imgnp,-1)
    cv2.imshow("test",img)
    cv2.waitKey('q')

Sarmentum answered 18/4, 2019 at 8:6 Comment(1)

Please try to correctly format your code if you're posting an answer - this isn't readable. Also, it doesn't really seem like you're addressing the question; you're using numpy, which isn't mentioned in the question, then seem to have some commented-out imports, and finally it seems to be an image you're downloading, which again isn't what the question asks for. – Bismuthous 18/4, 2019 at 8:31

Recommended topics

Hot tags