opening a url with urllib in python 3

Question

I'm trying to open the URL of this API from the sunlight foundation and return the data from the page in JSON. This is the code I've produced, minus the parenthesis around my API key.

import urllib.request.urlopen
import json

urllib.request.urlopen("https://sunlightlabs.github.io/congress/legislators?api_key='(myapikey)")

And I'm getting this error:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
ImportError: No module named request.urlopen

What am I doing wrong? I've looked into https://docs.python.org/3/library/urllib.request.html and still no progress.

Meshuga

Answer 1

You need to use from urllib.request import urlopen, also I suggest you use the with statement while opening a connection.

from urllib.request import urlopen

with urlopen("https://sunlightlabs.github.io/congress/legislators?api_key='(myapikey)") as conn:
    # dosomething

Unthrone

maybe i dont fully understand this concept yet im fairly new to python, but once i enter the code youve given i get an indentation error for line 4 (the empty line following as conn:) is there additional code i should be using to follow this? i dont wnat you to write my program for me, necessarily but any resources would be greatly appreciated. or gen. info – Meshuga

@BradleyD.Freeman-Bain: you can't have a with-statement without the following block. To avoid SyntaxError, add any code even mere pass will do. – Cocktail

@BradleyD.Freeman-Bain the "with" statement assigns the return value of urlopen to conn and allows you to operate with it in its own local scope directly below it. Once you escape it conn gets deleted. – Glare

Answer 2

In Python 3 You can implement that this way:

import urllib.request
u = urllib.request.urlopen("xxxx")#The url you want to open

Pay attention: Some IDE can import urllib(Spyder) directly, while some need to import urllib.request(PyCharm).

That's because you sometimes need to explicitly import the pieces you want, so the module doesn't need to load everything up when you just want a small part of it.

Hope this will help.

Lotta

Answer 3

from urllib.request import urlopen
from bs4 import BeautifulSoup

wiki = "https://en.wikipedia.org/wiki/List_of_state_and_union_territory_capitals_in_India"

page = urlopen(wiki)
soup =  BeautifulSoup(page, "html.parser" ).encode('UTF-8')

print (soup)

Surname

hi, if you post code, you can format it as code so that whitespaces are kept like you entered them and syntax highlighting is done. To do this select your code and click the curly braces in the editor, or indent the code with four spaces. I did that for you on this answer. Also it would be nice if you could write a little bit to explain your code. – Geum

Answer 4

urllib.request is a module whereas urlopen is a function. check out this link, it can help you clear your doubts. https://docs.python.org/3.0/library/urllib.request.html

Voluntary

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes – Akers

Answer 5

The problem is with your module import. There are multiple ways to do that in Python3.x:

from urllib.request import urlopen
urlopen('example.com/***')

or

from urllib import request
request.urlopen('example.com/***')

or

import urllib
urllib.request.urlopen('example.com/***')

Moreover, you can name the module you want to import as well:

import urllib.request as req  ## or whatever you want
req.urlopen('example.com/***')

You can use the one you prefer, but watch the sequence of modules and packages you are using.

Libove