Python request-html is not downloading Chromium

Question

import requests
from bs4 import BeautifulSoup
from requests_html import HTMLSession
url="https://dmarket.com/ingame-items/item-list/csgo-skins?title=recoil%20case"
sesion = HTMLSession()
response = sesion.get(url)
response.html.render()
soup = BeautifulSoup(response.html.html, features="html.parser")
print(soup)

After run it said

[INFO] Starting Chromium download.

After that crashes with this in VS Code:

Chromium downloadable not found at https://storage.googleapis.com/chromium-browser-snapshots/Win_x64/1181205/chrome-win.zip: Received <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: chromium-browser-snapshots/Win_x64/1181205/chrome-win.zip</Details></Error>

I tried installing different versions of requests_html

Elliellicott

Answer 1

UPDATE

Thanks to @Abdul Aziz Barkat's comment, it turns out that you can specify chromium version through environment variables, and pyppeteer will use it.

PYPPETEER_CHROMIUM_REVISION = '1263111'

---

requests-html uses pyppeteer library to download chromium, and it looks like version 1181205 of chromium which is hardcoded in pyppeteer has been removed from google storage.

Since requests-html installs pyppeteer with it, a simple workaround can be updating line 20 of pyppeteer's __init__.py file in your env:

__chromium_revision__ = '1181205' -> __chromium_revision__ = '1263111'

Note: I used version 1263111 because it's the latest for Win_x64 at the time of answering, and it works fine.

Proleg

You don't really need to change the packages code, it allows you to specify that version through environment variables – Colosseum

Answer 2

I tried the solution by @Lk4m1. This setup worked for me :

import asyncio
import os

PYPPETEER_CHROMIUM_REVISION = '1263111'

os.environ['PYPPETEER_CHROMIUM_REVISION'] = PYPPETEER_CHROMIUM_REVISION

from pyppeteer import launch


async def generate_pdf(url, pdf_path):
    browser = await launch()
    page = await browser.newPage()
    
    await page.goto(url)
    
    await page.pdf({'path': pdf_path, 'format': 'A4'})
    
    await browser.close()

# Run the function
asyncio.get_event_loop().run_until_complete(generate_pdf('https://example.com', 'example.pdf'))

Isopropanol

Answer 3

Use the following commands: 1-pip uninstall pyppeteer 2-pip install pyppeteer==1.0.0

it worked for me

Meridith

Answer 4

I had the same problem, and I fixed it by using an old version of PYPPETEER. The first time, I programmed a script with Linux and didn't encounter any issues. However, when I used another PC with Windows, I encountered the problem, so I resorted to using a different, older version. For example, it doesn't work with PYPPETEER version 2.0.0, but if I use PYPPETEER version 1.0.0, it functions perfectly well.

Waterer

Answer 5

I did just like @Lk4m1 said and it worked.

to find Python installation path 1- open cmd and type where python

find pyppeteer folder to find the init file C:\Users"Your User"\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyppeteer

Ennead

Answer 6

I had this problem too and I fixed it manually.
do this steps and it probably works.

1. Download a version of chromium from here with a Download manager (if you have 403 error try with VPN).

2. make this directory on Your computer: C:\Users\\(YoureUserName)\AppData\Local\pyppeteer\pyppeteer\Local-chromium\12606429(VresionOfChromium)

3. extract downloaded chromium in above directory address

notice: request-html looks for this file to find out chromium is installed or not:

C:\Users\\(YoureUserName)\AppData\Local\pyppeteer\pyppeteer\Local-chromium\12606429(VresionOfChromium)\chromium-win\chromium.exe

Musketeer

Answer 7

Modify Pyppeteer’s init.py File: Pyppeteer uses a hardcoded Chromium revision (e.g., 1181205) for downloading. To change this, you can update the chromium_revision variable in Pyppeteer’s init.py file. Locate the init.py file within your Pyppeteer installation (usually in the pyppeteer package folder). Change the value of chromium_revision from the existing revision (e.g., 1181205) to the desired revision (e.g., 1263111). Save the modified file.

Darindaring

Answer 8

If you want to hardcode it, that works too, but only if you change the code in the right place. Open the folder of the pyppeteer library and go into line 44 (if you use a win64 machine) of the chromium_downloader.py file.

windowsArchive = 'chrome-win'

downloadURLs = {
    'linux': f'{BASE_URL}/Linux_x64/{REVISION}/chrome-linux.zip',
    'mac': f'{BASE_URL}/Mac/{REVISION}/chrome-mac.zip',
    'win32': f'{BASE_URL}/Win/{REVISION}/{windowsArchive}.zip',
    'win64': f'{BASE_URL}/Win_x64/{REVISION}/{windowsArchive}.zip',
}

Replace the {REVISION} with 1263111 - restart you kernel (if you use Jupyter), execute again and it should work.

Do not change the variable above in line 32 - it'll cause problems with the requests-html library's AsyncHTMLSession module.

Phage

Sorry, my bad! Is it better like this? – Phage

Answer 9

I know it was not the same for you, but in my case instead of:

<Message>
    The specified key does not exist.
</Message>
<Details>
    No such object: chromium-browser-snapshots/Win_x64/1181205/chrome-win.zip
</Details>

the error message I got was:

<Message>
    Access denied.
</Message>
<Details>
    We're sorry, but this service is not available in your location
</Details>

This arose because my country blocks the use of Chromium.

For anyone with the same problem as me, just use a VPN, change your IPv4 DNS or anything that would get you through that and you will be good.

Foundry

Answer 10

If you're like me, and you're not an admin, and you had to install Python via Microsoft, then this solution is for you.

Changing a couple of values within the __init__ python file fixed both the op's issue, and the [WinError 14001] issue. Disclaimer: the WinError could be used generically for multiple issues, so I recommend checking the Event Viewer for your specific issue; to do this, launch the Event Viewer --> Expand the Windows Logs folder --> click Application --> click the Error log, and it should display an error message for either Pyppeteer or Chromium.

Without further ado, here's my solution:

1. Navigate to your C:\Users<your user name>\AppData\Local\ directory
2. Depending on how you installed Python, you'll want to open either the Python folder, the Pyppeteer folder, or the Packages folder
3. Search the folder/directory for the __init__ python file; it'll have a ".py" file extension, and should be located in the pyppeteer folder
4. Open the file, and do the following...
5. Find and copy __chromium_revision__ = '1181205'
6. Comment the line out by placing a hashtag in front of it; should like this: # __chromium_revision__ = '1181205'
7. Underneath it, we'll use a different version of chromium by typing the following: __chromium_revision__ = '1263111'
8. Save your changes, and keep the file open; the first change will solve the Chromium issue
9. Next, we're going to solve the [WinError 14001] issue by doing the following...
10. Find and copy __pyppeteer_home__ = os.environ.get('PYPPETEER_HOME', AppDirs('pyppeteer').user_data_dir) # type: str
11. Comment this line out just like we did in Step #4.ii
12. Underneath it, we'll point the variable to the TRUE location of our pyppeteer package by typing the following: __pyppeteer_home__ = 'C:\Users\<your user name>\AppData\Local\<the path of the pyppeteer folder where the init python file resides>'
13. Do NOT forget to use double back-slashes "\ \", or else the compiler will throw a Unicode exception; back-slashes are special escape characters (similar to Java and C# (C-Sharp)
14. Save your changes, close the file, and try running your script again. If it works, congrats!!! If not, don't give up, you're so close, just sleep on it, go out and touch grass, and then come back to it! You've got this!!!

Toady