Selenium Python: How to Scroll Down in a Pop Up window

Question

I am working on a Linkedin web scraping project. I am trying to get the list of companies that interest someone (notice I am not using the API). It is a dynamic website, so I would need to scroll down while scraping the names of the companies. I know how to do this in the MAIN window, but since Interest are a pop-up window this solution to scroll does not work. My code so far was:

from selenium.webdriver.common.keys import Keys
bar = driver.find_element_by_xpath('//ul[@class="entity-list row"]')
bar.send_keys(Keys.END)

Since it didn't work, I also tried:

bar = driver.find_element_by_xpath('//ul[@class="entity-list row"]')
driver.execute_script('arguments[0].scrollTop = arguments[0].scrollHeight', bar)

The problem is that I am not acting on the pop up window but in the main one so it has not the desired effect.

Reparation

Find the "scroll down" arrow css and click it until you find the element? That's how a human would do it. – Retarder

Yes, I agree with JeffC. I added my code so far to make it more complete. – Reparation

Answer 1

You can try to find element inside popup (the one that can be focused), for example some anchor:

element_inside_popup = driver.find_element_by_xpath('//div[@class="entity-list-wrapper ember-view"]//a')

and then use below code to scroll popup down:

from selenium.webdriver.common.keys import Keys

element_inside_popup.send_keys(Keys.END)

Luminance

Just a clarification: what does '//a' exactly do? I realized that without it, the code cannot focus the element. How would the driver know when the element is over so that it does not need to scroll down any more? – Reparation

//a means to return you the any first link on popup. Do you want to scroll to the very bottom of popup or to specific element? – Luminance

Thanks! I understand the '//a' magic now. I want to get the list of all companies and I realized this is only displayed fully if I get to the end of the element. – Reparation

Oh. I see. You need to make XHR while complete list not received? Try element_inside_popup.send_keys(Keys.END) in a while loop. You should check the count of companies and if on some iteration there is no new companies- break the loop – Luminance

I get this error selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <a class="class-small "> stale; either the element is no longer attached to the DOM or the document has been refreshed . any way to get arround it without time.sleep a bunch? – Bisque

I need to check your code. Can you create new ticket and describe your issue there? – Luminance

@Luminance Can you share the code with while loop and break statement to scroll the popup till end? – Vinyl

Answer 2

first I tried with this script but it doesn't work:

scrollable_popup = driver.find_element(By.XPATH, '/html/body/div[6]/div/div/div/div[2]')
for i in range(5):
    scrollable_popup.send_keys(Keys.END)
    time.sleep(2)

and then I used this script and it works fine with me:

scrollable_popup = driver.find_element(By.XPATH, '/html/body/div[6]/div/div/div/div[2]')
for i in range(5):
    driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scrollable_popup)
    time.sleep(2)

Sulk

Both is not working – Corson

The second one is working fine for me .. verify the XPATH of the popup – Sulk

Answer 3

I know this question is quite a few years old at this point. It does still come up in search on Google. I tried similar methods before similar to the answers in this thread. In my experience using Selenium, the JavaScript scroll method is great for getting dynamic content to load into a page. But it can be janky at times when trying to get specific elements to scroll into view. That could be from me not using JavaScript that often.

I ended up referring to the documentation Selenium provides and it looks like there is an element property that I didn't know about called location_once_scrolled_into_view which scrolls elements into view. It even works on those overlay popup windows. https://www.selenium.dev/selenium/docs/api/py/webdriver_remote/selenium.webdriver.remote.webelement.html#selenium.webdriver.remote.webelement.WebElement.location_once_scrolled_into_view

Arrive