I'm making a screen-scraper that collects schedule data from a TV channel website. To do that, I have to fetch the website's raw HTML. I'm trying my hand at unit-testing for the first time and I ha...

I am using BeautifulSoup to scrape an URL and I had the following code, to find the td tag whose class is 'empformbody': import urllib import urllib2 from BeautifulSoup import BeautifulSoup url = ...

Python 3. Probably need to use urllib to do this, I need to know how to send a request to a direct download link, and get the name of the file it attempts to save. (As an example, a KSP mod from ...

I'm using urlretrieve from the urllib module. I cannot seem to find how to add a User-Agent description to my requests. Is it possible with urlretrieve? or do I need to use another method?

In Python 3, the following code obtains the HTML source for a webpage. import urllib.request url = "https://docs.python.org/3.4/howto/urllib2.html" response = urllib.request.urlopen(url) response...

I'm trying to download all the PGNs from this site. I think I have to use urlopen to open each url and then use urlretrieve to download each pgn by accessing it from the download button near the bo...

First off, my problem is quite similar to this one. I would like a timeout of urllib.urlopen() to generate an exception that I can handle. Doesn't this fall under URLError? try: response = urlli...

I have seen this error on the forum and read through the responses yet I still don't understand what it is or how to address it. I'm scraping data from the internet from 16k links, my script scrape...

I'm new to Python and reading someone else's code: should urllib.urlopen() be followed by urllib.close()? Otherwise, one would leak connections, correct?

I am trying to download a pdf file from a website using urllib. This is what i got so far: import urllib def download_file(download_url): web_file = urllib.urlopen(download_url) local_file = op...

I am trying to make a POST request to the following page: http://search.cpsa.ca/PhysicianSearch In order to simulate clicking the 'Search' button without filling out any of the form, which adds dat...

I am using the following code to validate the ssl certificate status, Can we get more details about certificate like common name (CN),expiry date and issuer using request module or urllib import ...

How can I avoid exceptions from urllib.request.urlopen if response.status_code is not 200? Now it raise URLError or HTTPError based on request status. Is there any other way to make request with p...

raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='mycompanyurl.in', port=443): Max retries exceeded with url: /api/v1/issues.json (Caused by ...

I would like to use urllib.quote(). But python (python3) is not finding the module. Suppose, I have this line of code: print(urllib.quote("châteu", safe='')) How do I import urllib.quote? impor...

I'm trying to open the URL of this API from the sunlight foundation and return the data from the page in JSON. This is the code I've produced, minus the parenthesis around my API key. import urllib...

CODE: import networkx as net from urllib.request import urlopen def read_lj_friends(g, name): # fetch the friend-list from LiveJournal response=urllib.urlopen('http://www.livejournal.com/misc/fdat...

I am building a script to download and parse benefits information for health insurance plans on Obamacare exchanges. Part of this requires downloading and parsing the plan benefit JSON files from e...

I am trying to send a request wtih post method to an API, my code looks like the following one: import urllib.request import json url = "https://api.cloudflareclient.com/v0a745/reg" refe...

Which URL parsing function pair should I be using and why? urlparse and urlunparse, or urlsplit and urlunsplit?

In Python, what are the differences between the urllib, urllib2, urllib3 and requests modules? Why are there three? They seem to do the same thing...

I'm trying to search the world wide web for this answer, but I feel there answer may be no. I'm using Python 3.5 and a library called urllib.request with a method called urllib.request.urlopen(url)...

I have been using urllib.request.FancyURLOpener() to retrieve images from URLs. My code is something like this: class MyOpener(FancyURLopener): version = "Mozilla/5.0 (Windows; U; Windows NT 5.1;...

(Python 3.4.2) I've got a weird error when I run 'urllib.request.urlopen(url)' inside of a script. If I run it directly in the Python interpreter, it works fine, but not when I run it inside of a s...

python 3.5.2 code 1 import urllib s = urllib.parse.quote('"') print(s) it gave this error: AttributeError: module 'urllib' has no attribute 'parse' code 2 from urllib.parse import quote...