Make headless browser stop loading page

Question

I am using the watir-webdriver ruby gem. It starts the browser (Chrome) and begins to load a page. The page is loading too slowly, and watir-webdriver raises a timeout error. How can I make the browser stop loading the page?

require 'watir-webdriver'

client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 10
@browser = Watir::Browser.new :chrome, :http_client => client

sites = [
  "http://google.com/",
  "http://yahoo.com/",
  "http://www.nst.com.my/", # => This is the SLOW site
  "http://drupal.org/",
  "http://www.msn.com/",
  "http://stackoverflow.com/"
]

sites.each do |url|

  begin
    @browser.goto(url)
    puts "Success #{url}"
  rescue
    puts "Timeout #{url}"
  end

end

########## Execution result ########## 

# Success http://google.com/
# Success http://yahoo.com/
# Timeout http://www.nst.com.my/
# Timeout http://drupal.org/
# Timeout http://www.msn.com/
# Timeout http://stackoverflow.com/

########## Expected result ########## 

# Success http://google.com/
# Success http://yahoo.com/
# Timeout http://www.nst.com.my/
# Success http://drupal.org/
# Success http://www.msn.com/
# Success http://stackoverflow.com/

It looks like the browser doesn't respond to any other commands before it finishes loading the page. How can I force the browser to throw away the page it is loading and execute the next command?

UPDATED

I have found an interesting capability flag loadAsync http://src.chromium.org/svn/trunk/src/chrome/test/webdriver/webdriver_capabilities_parser.cc Maybe it can be useful for solving this problem? I don't understand yet how to make watir (webdriver) to set this when starting the chromedriver. This flag was introduced here http://codereview.chromium.org/7582005/

Titivate

At the top of my head there is only one explanation for this thing. The Watir is single threaded. – Cenogenesis

I am not trying to use it in multiple threads. I just wonder why after one command and its timeout is impossible to execute next command. – Titivate

When i say single threaded i mean that when sending a comand to webdriver to do something you a synchronous call and can't do anything until gets the response from webdriver. So the thing you want to do in this scenario cannot be done, until you change synchronous call to asynchronous one. – Cenogenesis

It looks like several people have run into this issue too - see chromedriver issue 9, but it has not yet been resolved. – Rife

Use firefox for this test instead of chrome? – Royster

The Firefox (driver?) has the same issue and produces the same results. – Titivate

Answer 1

There are a couple of different ways to do what you're wanting, but here's what I would do:

require 'watir-webdriver'

client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 60 
@browser = Watir::Browser.new :firefox, :http_client => client

begin
  @browser.goto mySite
rescue => e
  puts "Browser timed out: #{e}"
end

next_command

If you have a lot of sites you're trying to load for confirmation of timeout or not, put them in an array:

mySites = [
          mySite1,
          mySite2,
          mySite3
]

mySites.each do |site|
   begin
      @browser.goto site
   rescue
      "Puts #{site} failed to load within the time allotted."
   end
end

UPDATE for proof of concept. This script always proceeds to step 2. The rescue isn't even necessary for the second goto, but is being used for clearer output. How is your script different than this?

require 'watir-webdriver'

client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 1     #VERY low timeout to make most sites fail
@browser = Watir::Browser.new :firefox, :http_client => client


def testing
   begin
     @browser.goto "http://www.cnn.com"
   rescue => e
     puts "Browser timed out on the first example"
   end

   begin
     @browser.goto "http://www.foxnews.com"
   rescue => e
     puts "Browser timed out on the second example"
   end
end

Racoon

The problem is that any "next_command" (like a @browser.goto "another-site.com" or @browser.close) will raise another Timeout error. It looks like the browser is "frozen" after raising Timeout for the first time. – Titivate

@Titivate I have tried this a number of times and I am unable to reproduce your error. I have added a code example to show you what I am trying, and it works as expected. – Racoon

Here is my testing code pastebin.com/vNiKZ9tP It confirms described behavior. I wonder why the script reachs the second rescue block. It differs from your code that only first site should reach the timeout. – Titivate

I'm not sure if the sleep(999) statement on your local server would change anything at all, but I cannot reproduce that part exactly. Have you tried running it against two known external websites with the timeout set to "1" as I have it? Also, what version of watir-webdriver and selenium-webdriver are you using? ruby 1.9.3p125 (2012-02-16) [i386-mingw32] watir-webdriver (0.5.3) selenium-webdriver (2.20.0) – Racoon

I have run your code, but it doesnt show the actual problem. Here is a realworld example pastebin.com/0vSywwYE . Expected results are different from the actual ones. ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-darwin11.3.0] selenium-webdriver (2.20.0) watir-webdriver (0.5.4) – Titivate

You are correct - I pasted that same code and was able to reproduce your results even after a few tweaks. Let's see what other attention we can get for this thread! – Racoon

Adam, your code is using Firefox, his is using Chrome. Are you seeing the same behavior with both browsers, or is this issue only occurring with chrome? – Royster

@ChuckvanderLinden I have tested with both. when I manually clicked "stop" in the browser to stop www.nst.com.my from perpetually loading, I noticed that the next two results failed but the last passed. Doing some simple math - since the client timeout is set to 10 seconds, I waited 30+ seconds before clicking the stop button. The "clock" had continued to count using up the 10 seconds allotted for the next two sites, but the final site passed since it loaded in under the 10 seconds left over. Does that make sense? That's why the problematic site causes the rest to fail. – Racoon

So if you have three sites: problem site, fast site and fast site 2... and you set the timeout to 10 seconds. If the first site takes > 30 seconds to load, all three will fail. If the first site takes 20 seconds to load, the first two sites will show failed, but the last site may pass (if it is able to respond in under the 10 seconds remaining). That said, I don't know how to 'break' the wait while the first slow site is loading past the 10-second timeout mark. – Racoon

@ChuckvanderLinden This problem belongs to both browsers (Chrome and Firefox). adam-reed You are right, that when I manually click the sopt button in browser window, it "unfreeze" the browser for webdriver and it goes on processing next commands (sites). The example SLOW site "www.nst.com.my" may be not enough slow for experiment purposes. I even tried to send escape key to browser, but it is not respontable for this command (browser.send_keys) too. Even more, I looked for some Chrome setting, like page load timeout but didnt find anything. – Titivate

@AdamReed, I have updated the question, the capabilities flag loadAsync. What do you know about it? – Titivate

Answer 2

You can stop Google loading the page using AutoIT to send the Escape key. This is similar to what you originally tried to do, but using AutoIT directly rather than through the broken Watir::Browser object.

require 'watir-webdriver'
require 'win32ole'

client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 5
@browser = Watir::Browser.new :chrome, :http_client => client
begin
    @browser.goto "http://www.nst.com.my/"
rescue
    autoit = WIN32OLE.new("AutoItX3.Control")
    autoit.AutoItSetOption("WinTitleMatchMode", 2) 
    autoit.WinActivate("Google")
    autoit.Send("{ESC}")
end
@browser.goto "http://www.google.ca"

Note: I tried to get autoit.ControlSend("Google", "", "", "{ESC}") to work so that it would not need the browser to be the active window. While it worked when run by itself, for some reason I could never get it to work in the above script (ie key was sent but browser did not react as expected).

Rife

Everything is going on Linux machine in headless mode (production). – Titivate

Answer 3

I've been fighting with this problem for awhile, I know that this post is from 2012 but I still haven't found anything that solves this issue.. So i made a work around.

require 'watir-webdriver'

client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 5
@browser = Watir::Browser.new :firefox, :http_client => client

@browser.goto "twitter.com"

#after the page loads, log in 
def test    
  begin
     temp = []
     temp = @browser.cookies.to_a
     @browser.goto "twitter.com:81"
  rescue => e
     puts "Browser timed out"
     @browser.close
     @browser = Watir::Browser.start "twitter.com"

     temp.each do |me|
       @browser.cookies.add(me[:name], me[:value])
     end
     @browser.refresh 

  end
end

The extra code that saves and restores your cookies will allow you to stay logged into sites you're using. This sucks but its the only work around I can think of. Again this was back in 2012, so if anyone finds anything that works better please correct me.

Carrillo

It doesnt resolve the problem, because your @browser.close won't work. – Titivate

On my computer it does. I'm using Linux debian 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u1 x86_64 GNU/Linux, with Firefox 17.0.10, watir 5, watir-webdriver 0.6.4 – Carrillo

ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux] – Carrillo

Answer 4

I have had the problem of timeouts as well. My understanding from online research is that after Selenium WebDriver encounters a timeout, it gets into bad shape and misbehaves (multithreading issue). I have followed the advices found here:

https://github.com/watir/watir-webdriver/issues/137

I implemented active killing (instead of browser.close) and restarting of the Watir::Browser after any exception rescue.

Corsair