urllib.error.URLError: <urlopen error unknown url type: https>

Question

Hello I am trying to learn web scraping. I installed Anaconda3 in Windows 10. Conda version 4.5.12. Python version 3.7.1.

I wrote following script which produces the mentioned error.

import bs4
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as request
with request('https://google.com') as response:
    page_html = response.read()
page_soup = soup(page_html, "html.parser")
print(page_soup)

The error was from this line:

with request('https://google.com') as response:
...
...
raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: https>

However, when I opened up my Anaconda prompt and executed line by line from above script it worked flawlessly.
Can Anyone help me understand what went wrong? How can I make the script run from the console without getting this error?

Enfeeble

Answer 1

I solved the problem by reinstalling the anaconda library. On the installation process I chose to set up the PATH variable although it was not recommended. ( So i did not set the PATH first time ) So, basically I ignored the recommendation. And After that it worked. I do not know exactly what caused the problem first time. But now it is working.

Enfeeble

Answer 2

I had a similar problem installing the emsdk with Anaconda installed with the recommended settings.

I solved this way:

Open Anaconda Navigator > Click Environments > Select base(root) > Click the play icon > Select Open Terminal > Navigate to the script folder > Run the script

Reflectance

Answer 3

May be this works,

response= request('https://google.com')
page_html = response.read() if response else ""
page_soup = soup(page_html, "html.parser")
print(page_soup)

Marrano

Thank you for your response, but the problem does not resolved. – Enfeeble

Answer 4

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

This is the way to allow unverified SSL

Armchair

Thank you for your response but this throws another error "import _ssl # if we can't import it, let the error propagate ImportError: DLL load failed: The specified module could not be found." was the error message – Enfeeble

Answer 5

Try the below code:

import bs4
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen, Request
response = urlopen('https://www.google.com')
page_html = response.read()
page_soup = soup(page_html, "html.parser")
print(page_soup)

Alyosha

Thank you for your response. I made it work by reinstalling and chose set Path although it was not recommended during the anaconda installation process. Then the same code works just fine. – Enfeeble

Answer 6

I was facing this problem as well. So what happened and what was the solution in my case.

Since my SSD one day had a I/O error I needed to buy a new SSD and install Windows 10 Pro. After Windows I was installing Python 3.11.2 and try to run a simple urllib code.

import urllib.request
 
web=urllib.request.urlopen('https//exemple.com')
....

I got thist error:

AttributeError: partially initialized module 'ssl' has no attribute 'create_default_context' (most likely due to a circular import)

After some google research I did many solution to solve this problem and at the end problem was just simple Windows Update since I didn't perform update after fresh installation... Ok, after that I did Windows Update but now I was receiving error which is the subject of this question topic...

Error was:

    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: urlopen error unknown url type: https

So I have tried everything, from different python versions and different python libraries, since Windows was not a problem, program just worked on another computer with Win 10 Pro, I have suspected drivers, installed all drivers and updated but problem was the same...

So I had a same program on USB, I have insert the USB and try the same program on USB it just worked without a problem... and I have started to analyze the problem. I have made another directory save the python program there and it worked perfectly, but in the previous directory with all other files I was getting the same error, so I have suspected that some file within that directory is making problem.

I have found out that by trying different solutions I have made a file name ssl.py and because of that i was getting the error described above.

After renaming the file to ssl1.py, my urllib script just worked fine...

Oaxaca