Python Selenium take screenshots and Save as PDF for windows opened with document.write()

Question

I'm using Selenium with Python (in Jupyter notebook). I have a number of tabs open, say 5 tabs (with all elements already finished loading) and I would like to cycle through them and do 2 things:

1. Take a screenshot,
2. (As a bonus) Print each to PDF using Chrome's built-in Save as PDF function using A4 landscape, normal scaling, and a specified default directory, with no user interaction required.

(In the code below I focus on the screenshot requirement, but would also very much like to know how to Save it as a PDF)

This code enables looping through the tabs:

numTabs = len(driver.window_handles)
for x in range(numTabs):
    driver.switch_to.window(driver.window_handles[x])    
    time.sleep(0.5)     

However, if I try to add a driver.save_screenshot() call as shown below, the code seems to halt after taking the first screenshot. Specifically, '0.png' is created for the first tab (with index 0) and it switches to the next tab (with index 1) but stops processing further. It doesn't even cycle to the next tab.

numTabs = len(driver.window_handles)
for x in range(numTabs):
    driver.switch_to.window(driver.window_handles[x])    
    driver.save_screenshot(str(x) + '.png') #screenshot call
    time.sleep(0.5)    

Edit1: I modified the code as shown below to start taking the screenshots from window_handles[1] instead of [0] as I don't really need the screenshot from [0], but now not even a single screenshot is generated. So it seems that the save_screenshot() call doesn't work after even the initial switch_to.window() call.

tabs = driver.window_handles
for t in range(1, len(tabs)):
    print("Processing tab " + tabs[t]) 
    driver.switch_to.window(tabs[t])  
    driver.save_screenshot(str(t) + '.png') #screenshot call, but the code hangs. No screenshot taking, no further cycling through tabs.

Edit2: I've found out why my code is "hanging", no matter which method of printing to PDF or taking screenshots I'm using. I mentioned earlier that the new tabs were opened via clicking on buttons from the main page, but upon closer inspection, I see now that the content of the new tabs is generated using document. write(). There's some ajax code that retrieves waybillHTML content that is then written into a new window using document.write(waybillHTML)

For more context, this is an orders system, with the main page with a listing of orders, and a button next to each order that opens up a new tab with a waybill. And the important part is that the waybills are actually generated using document. write() triggered by the button clicks. I notice that the "View page source" option is greyed out when right-clicking in the new tabs. When I use switch_to.window() to switch to one of these tabs, the Page.printToPDF times out after 300 (seconds I suppose).

---------------------------------------------------------------------------
TimeoutException                          Traceback (most recent call last)
<ipython-input-5-d2f601d387b4> in <module>
     14     driver.switch_to.window(handles[x])
     15     time.sleep(2)
---> 16     data = driver.execute_cdp_cmd("Page.printToPDF", printParams)
     17     with open(str(x) + '.pdf', 'wb') as file:
     18         file.write(base64.b64decode(data['data']))

...

TimeoutException: Message: timeout: Timed out receiving a message from renderer: 300.000
  (Session info: headless chrome=96.0.4664.110)

So my refined question should be how to use Page.printToPDF to print a page in a new window (that is generated dynamically with document. write()) without timing out?

One approach I tried was to do this:

from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities().CHROME
caps["pageLoadStrategy"] = "none"
driver = webdriver.Chrome(options=chrome_options, desired_capabilities=caps)

referring to: this question

but the problem is this is too 'aggressive' and prevents the code from logging into the ordering system and doing the navigating & filtering necessary to get the browser to the point where I get the orders listing page with the waybill generating buttons (i.e. the original setup in this question).

Edit3: At this point, I've tried something as simple as just getting the page source

try:
    pageSrc = driver.find_element(By.XPATH, "//*").get_attribute("outerHTML")
    print(pageSrc)

of the dynamically generated tabs (long after they had completed rendering and I can see the content on the screen (not using headless for this stage of the debugging)) and even this itself is throwing a TimeoutException, so I don't think it's an issue with waiting for content to load. Somehow the driver is unable to see the content. It might be something peculiar with how these pages are generated - I don't know. All the methods suggested in the answers for taking screenshots and saving PDFs are good I'm sure for otherwise normal windows. With Chrome the View page source remains greyed out, but I can see regular HTML content using Inspect.

Edit4: Using Chrome's Inspection function, the page source of the dynamically generated page has this HTML structure:

Curiously, "View page source" remains greyed out even after I'm able to Inspect the contents:

Edit5: Finally figured it out. When I clicked on the button for generating the new tab, the site would make an ajax call to fetch the HTML content of the new page, and document.write it to a new tab. I suppose this is outside the scope of selenium to handle. Instead, I imported selenium-wire, used driver.wait_for_request to intercept the ajax call, parsed the response containing the HTML code of the new tab, and dumped the cleaned HTML into a new file. From then on generating the PDF can be easily handled using a number of ways as others have already suggested.

Jannelle

My Chrome (Version 96.0.4664.11) does not have a "Save as PDF" option. Why does yours? – Rossen

Not sure why yours doesn't, but I've been using that function for years. At least this other person seems to have this function: support.google.com/chrome/thread/87680283/… – Jannelle

My error -- I missed where you said it was under the Print menu. – Rossen

@stackoverflow.com/users/773694/fortunerice pdfkit – Verein

pip install pdfkit – Verein

I was wondering it you were ever able to get the pages in question to load their content for printing? – Befitting

No still not yet. The content renders on screen almost immediately after opening up in a new tab, but to the webdriver the content still never loads (times out after 300 seconds). If I ever figure it out I'll post it back here. – Jannelle

@Lifeiscomplex please see my Edit5 - I figured it out in the end. – Jannelle

Answer 1

As your usecase is to Print each to PDF using Chrome's built in Save as PDF function or take a screen shot, instead of opening all the additional links at the same time you may like to open the links in the adjascent tab one by one and take the screenshot using the following Locator Strategies:

• Code Block:

  num_tabs_to_open = len(elements_href)
  windows_before  = driver.current_window_handle
  # open the links in the adjascent tab one by one to take screenshot
  for href in elements_href:
      i = 0
      driver.execute_script("window.open('" + href +"');")
      windows_after = driver.window_handles
      new_window = [x for x in windows_after if x != windows_before][0]
      driver.switch_to.window(new_window)
      driver.save_screenshot(f"image_{str(i)}.png")
      driver.close()
      driver.switch_to.window(windows_before)
      i = i+1
  

---

References

You can find a relevant detailed discussion in:

• InvalidSessionIdException: Message: invalid session id taking screenshots in a loop using Selenium and Python

Kiely

Thanks but I don't have the individual hrefs. All the tabs are opened up by clicking on a button in the main tab (the listings tab) that opens up new tabs in about:blank. I have no problems cycling through the tabs, but the save_screenshot call only works the first time afterwards it just seems to hang and doesn't switch tabs further more than once. – Jannelle

The issue is the complexity of handling 5 tabs and you may like to avoid it. If the elements are <button> still the approach remains the same. Instead of driver.execute_script() you have to button.click() – Kiely

Please see my Edit2 for a refining of the question. – Jannelle

Answer 2

Update 2

Based on your recent Edit 3, I now get the source for the new window by retrieving it with an AJAX call. The main page that the driver gets is:

test.html

<!doctype html>
<html>
<head>
<meta name=viewport content="width=device-width,initial-scale=1">
<meta charset="utf-8">
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script>
<script>
function makeRequest() {
    var req = jQuery.ajax({
        'method': 'GET',
        'url': 'http://localhost/Booboo/test/testa.html'
        });
    req.done(function(html) {
        let w = window.open('', '_blank');
        w.document.write(html);
        w.document.close();
    });
}
</script>
<body>
</body>
<script>
$(function() {
   makeRequest();
});
</script>
</html>

And the document it is retrieving, testa.html, as the source for the new window is:

testa.html

<!doctype html>
<html>
<head>
<meta name=viewport content="width=device-width,initial-scale=1">
<meta charset="utf-8">
</head>
<body>
<h1>It works!</h1>
</body>
</html>

And finally the Selenium program gets test.html and enters a loop until it detects that there are now two windows. It then retrieves the source of the second window and takes a snapshot as before using Pillow and image2Pdf.

from selenium import webdriver
import time

def save_snapshot_as_PDF(filepath):
    """
    Take a snapshot of the current window and save it as filepath.
    """
    from PIL import Image
    import image2pdf
    from tempfile import mkstemp
    import os

    if not filepath.lower().endswith('.pdf'):
        raise ValueError(f'Invalid or missing filetype for the filepath argument: {filepath}')

    # Get a temporary file for the png
    (fd, file_name) = mkstemp(suffix='.png')
    os.close(fd)
    driver.save_screenshot(file_name)
    img = Image.open(file_name)
    # Remove alpha channel, which image2pdf cannot handle:
    background = Image.new('RGB', img.size, (255, 255, 255))
    background.paste(img, mask=img.split()[3])
    background.save(file_name, img.format)
    # Now convert it to a PDF:
    with open(filepath, 'wb') as f:
        f.write(image2pdf.convert([file_name]))
    os.unlink(file_name) # delete temporary file


options = webdriver.ChromeOptions()
options.add_argument("headless")
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)

try:
    driver.get('http://localhost/Booboo/test/test.html')
    trials = 10
    while trials > 10 and len(driver.window_handles) < 2:
        time.sleep(.1)
        trials -= 1
    if len(driver.window_handles) < 2:
        raise Exception("Couldn't open new window.")
    driver.switch_to.window(driver.window_handles[1])
    print(driver.page_source)
    save_snapshot_as_PDF('test.pdf')
finally:
    driver.quit()

Prints:

<html><head>
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta charset="utf-8">
</head>
<body>
<h1>It works!</h1>


</body></html>

Rossen

Thanks I'll give this a shot too but my current problem is actually getting stuck on the driver.save_screenshot(file_name) call. After I call it, it just hangs. Good to know about the img2pdf though. – Jannelle

Please see my Edit2. I've found that it wasn't 'hanging' it was timing out. – Jannelle

I have updated the demo to use document.write() to create the new window and there is no hanging. The only difference is that there is no AJAX involved but that only determines as I understand it where the string that is being used as the argument to the call to write comes from, so it is difficult to see how that would make a difference. – Rossen

I have updated the demo again based this time using AJAX to retrieve the source for the new window. So I am trying to emulate as closely as possible the same scenario you have. I also am running the same version of Chrome as you. Again, I have no problems. – Rossen

Answer 3

I'm a little confused on these part of your questions:

Take a screen shot,

(As a bonus) Print each to PDF using Chrome's built in Save as PDF function using A4 landscape, normal scaling, and a specified default directory, with no user interaction required.

The function save_screenshot saves an image file to your file system. To convert this image file to a PDF you would have to open it and write it out to PDF file.

That task is easy, using various Python PDF modules. I have code for this so let me know if you need it and I will add it the code below.

Concerning printing the webpages in the tabs as a PDFs you would use execute_cdp_cmd with Page.printToPDF. The code below can be modified to support your unknown urls scenario via the button clicks. If you need help with this let me know.

import base64
import traceback
from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import TimeoutException

chrome_options = Options()
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument("--disable-infobars")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-popup-blocking")

# headless mode is required for this method of printing
chrome_options.add_argument("--headless")

# disable the banner "Chrome is being controlled by automated test software"
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches", ['enable-automation'])

driver = webdriver.Chrome('/usr/local/bin/chromedriver', options=chrome_options)

# replace this code with your button code
###
driver.get('https://abcnews.go.com')

urls = ['https://www.bbc.com/news', 'https://www.cbsnews.com/', 'https://www.cnn.com', 'https://www.newsweek.com']
for url in urls:
    driver.execute_script(f"window.open('{url}','_blank')")
    # I'm using a sleep statement, which can be replaced with 
    # driver.implicitly_wait(x_seconds) or even a 
    # driver.set_page_load_timeout(x_seconds) statement
    sleep(5)
###


# A4 print parameters
params = {'landscape': False,
          'paperWidth': 8.27,
          'paperHeight': 11.69}

# get the open window handles, which in this care is 5
handles = driver.window_handles
size = len(handles)

# loop through the handles 
for x in range(size):
    try: 
       driver.switch_to.window(handles[x])
       # adjust the sleep statement as needed
       # you can also replace the sleep with 
       # driver.implicitly_wait(x_seconds)
       sleep(2)
       data = driver.execute_cdp_cmd("Page.printToPDF", params)
       with open(f'file_name_{x}.pdf', 'wb') as file:
          file.write(base64.b64decode(data['data']))
       # adjust the sleep statement as needed
       sleep(3)
     except TimeoutException as error:
        print('something went wrong')
        print(''.join(traceback.format_tb(error.__traceback__)))

driver.close()
driver.quit()

Here is one of my previous answers that might be useful:

• Selenium print PDF in A4 format

Befitting

Thanks I wasn't aware that there was a Page.printToPDF function. Found the complete docs here chromedevtools.github.io/devtools-protocol/tot/Page/…. I'll give it a try and get back. Also to clarify my question, my ultimate objective is to Save as PDF (preferably with the text selectable, so not "hard-burned"), but as a fallback the minimum requirement is to take a screenshot if the PDF printing causes too many problems. – Jannelle

Selenium has a lot of features with limited documentation. If you have any issues let me know. – Befitting

What do you mean having the text selectable? Via OCR or do you want to extract the text in some way? If it is the latter please look through my previous answers for ways to do this. – Befitting

The text is selectable using Page.printToPDF so no need to worry about that. But please see my Edit2 for a refining of the question. – Jannelle

Have you tried to add a driver.implicitly_wait(x_seconds) statement in the loop through the handles code? This might help with the TimeoutException issues. – Befitting

You can also try using a set_page_load_timeout(x_seconds) to make sure a page is completely rendered. – Befitting

I wrapped the handle loop with a try/catch block. – Befitting

Thanks I tried adding waits but there's something peculiar with the way those tabs are generated and it still times out. (Please see Edit3) – Jannelle

Based on your latest update it sounds like the content is being created in something outside of the current driver window. I have seen this happen when an iframe is involved. Here is something that should help determine this: guru99.com/handling-iframes-selenium.html – Befitting

Just checked - not an iframe. I added a screenshot of the structure of page that is generated (the one I'm trying to Print to PDF). Please see Edit4. – Jannelle

This is hard to troubleshoot. I assume that you cannot share the URL of the site. If this is true then can you share any specific details about the site? – Befitting