webdriver wait for ajax request in python

Question

Currently I am writing webdriver test for search which uses ajax for suggestions. Test works well if I add explicit wait after typing the search content and before pressing enter.

wd.find_element_by_xpath("//div[@class='searchbox']/input").send_keys("obama")
time.sleep(2)
wd.find_element_by_xpath("//div[@class='searchbox']/input").send_keys(Keys.RETURN)

but

wd.find_element_by_xpath("//div[@class='searchbox']/input").send_keys("obama")
wd.find_element_by_xpath("//div[@class='searchbox']/input").send_keys(Keys.RETURN)

fails. I am running tests on ec2 with 1 virtual cpu. I am suspecting, I pressed enter even before GET requests related to search are sent and if I press enter before suggestions, it fails.

Is there any better way that adding explicit waits?

Irrefutable

Answer 1

You indeed can add an explicit wait for the presence of an element like

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0

ff = webdriver.Firefox()
ff.get("http://somedomain/url_that_delays_loading")
ff.find_element_by_xpath("//div[@class='searchbox']/input").send_keys("obama")

try:
    element = WebDriverWait(ff, 10).until(EC.presence_of_element_located((By.ID, "keywordSuggestion")))
finally:
    ff.find_element_by_xpath("//div[@class='searchbox']/input").send_keys(Keys.RETURN)
    ff.quit()

See: http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp#explicit-and-implicit-waits

Webworm

Answer 2

Add this method, where I ensure the API responses are back from server

def wait_for_ajax(driver):
    wait = WebDriverWait(driver, 15)
    try:
        wait.until(lambda driver: driver.execute_script('return jQuery.active') == 0)
        wait.until(lambda driver: driver.execute_script('return document.readyState') == 'complete')
    except Exception as e:
        pass

Osteotomy

In general, bare except statements are not a good practice. Why do you need the try...except... blocks here? – Impolite

@Impolite it depends on your need; you can remove it if not needed. – Osteotomy

Answer 3

You can use selenium-wire instead which is a wrapper of selenium. It will setup a proxy with port 8087 in client for webdriver and listen all the requests. The code is like

from seleniumwire import webdriver

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=127.0.0.1:8087')
driver = webdriver.Remote(
    command_executor="http://127.0.0.1:4444/wd/hub",
    options=chrome_options,
    seleniumwire_options={
        'auto_config': False,
        'port': 8087,
        'addr': '127.0.0.1',
    }
)

Snobbery

Answer 4

And what about:

    driver.implicitly_wait(10)

for your example:

    wd.implicitly_wait(10)

In this case every time you are going to click or find element driver will try to do this action every 0.5 second during 10 seconds. In this case you don't need to add wait every time. Note: But it is only about element on screen. It will not wait until some JS actions ends.

Two