Can a watir browser object be re-used in a later Ruby process?

Question

So let's say pretty often a script runs that opens a browser and does web things:

require 'watir-webdriver'

$browser = Watir::Browser.new(:firefox, :profile => "botmode")
 => #<Watir::Browser:0x7fc97b06f558 url="about:blank" title="about:blank"> 

It could end gracefully with a browser.close, or it could crash sooner and leave behind the memory-hungry Firefox process, unnoticed until they accumulate and slow the server to a crawl.

My question is twofold:

• What is a good practice to ensure that even in case of script failure anywhere leading to immediate error exit, the subprocess will always get cleaned up (I already have lots of short begin-rescue-end blocks peppered for other unrelated small tests)
• More importantly, can I simply remember this Watir::Browser:0x7fc97b06f558 object address or PID somehow and re-assign it to another $browser variable in a whole new Ruby process, for example irb? I.e. can an orphaned browser on webdriver be re-attached later in another program using watir-webdriver on the same machine? From irb I could then get in and re-attach to the browser left behind by the crashed Ruby script, to examine the website it was on, check what went wrong, what elements are different than expected, etc.

Another hugely advantageous use of the latter would be to avoid the overhead of potentially hundreds of browser startups and shutdowns per day...best to keep one alive as sort of a daemon. The first run would attempt to reuse a previous browser object using my specially prepared botmode profile, otherwise create one. Then I would deliberately not call $browser.close at the end of my script. If nothing else I run an at job to kill the Xvfb :99 display FF runs inside of at the end of the day anyway (giving FF no choice but to die with it, if still running). Yes I am aware of Selenium standalone jar, but trying to avoid that java service footprint too.

Apologies if this is more a basic Ruby question. I just wasn't sure how to phrase it and keep getting irrelevant search results.

Novena

Is this something where it is important the code interact with an actual browser like FF? If speed is the issue then maybe consider using the headless option instead of FF? or a browser like Chrome or Opera that starts up faster? If we knew a bit more about what you were trying to accomplish we might be able to offer better advice and/or solutions – Hinda

In favor of speed and esp. limiting memory consumption,Yes I had hoped to avoid a browser altogether, but trying Htmlunit +Javascript enabled in the past resulted in catastrophic failures or too many coding differences....is that what you had in mind? If it weren't for mandatory AJAX support I'd stick to Mechanize not Watir like I do for TDAmeritrade.com, magnitudes faster and lighter – Novena

Yes I was thinking about the HtmlUnit route as described in this blog posting watirmelon.com/2010/12/14/… But if you have Ajax or other issues that require a real browser then yes you'd want to be using FF or Chrome etc. For that I think the answer I provided below would probably apply – Hinda

Answer 1

I guess, U cant just remember the variable from another process. But the solution might be creating a master process and process your script in loop in thread, periodically checking the browser running state. I'm using some thing similar in my acceptance tests on Cucumber + watir. So it will be some thing like that:

require 'rubygems'
require 'firewatir' # or watir
@browser = FireWatir::Firefox.new

t = Thread.new do
    @browser.goto "http://google.com"
    #call more browser actions here
end
while not_exit?
  if t.stop?
      # error occurred in thread, restart or exit
  end
  if browser_live?
      # browser was killed for a some reason
      # restart or exit
  end
end
@browser.close

not_exit? - can be over TRAP for the ctrl+C

browser_live? - you can check if firefox browser running with processes listings

It is quite tricky but might work for you

Huckster

Thanks, somewhat along those lines, I am considering daemonizing my low-level watir script, though I'm afraid that would require an overhaul in the higher levels wrapping around it using its current straightforward command-line API. – Novena

Figured it might not be a bad idea to create a separate question to explore the strategy of creating a simple, separate daemon whose only purpose is to keep alive persistent Ruby objects shared to my main Ruby script – Novena

I could extend my answer with that – Huckster

Answer 2

You can use DRb like this:
browsers pool:

require 'drb'
require 'watir'

browser = Watir::Browser.new :chrome
DRb.start_service 'druby://127.0.0.1:9395', browser

gets

and then from test script use this browser:

require 'drb'
browser = DRbObject.new_with_uri 'druby://127.0.0.1:9395'
browser.goto 'stackoverflow.com'

Chrissie

Answer 3

I'm pretty sure that at the point ruby exits, any handles or pointers to something like a browser object would become invalid. So re-using something in a later ruby process is likely not a good approach. In addition I might be wrong on this, but it does seem that webdriver is not very good at connecting to a running browser process. So for your approach to work it would really all need to be wrapped by some master process that was calling all the tests etc.. and hey wait a sec, that's starting to sound like a framework, which you might already (or perhaps should be) using in the first place.

So a better solution is probably to look at whatever framework you are using to run your tests and investigate any capability for 'setup/teardown' actions (which can go by different names) which are run before and after either each test, groups of tests, or all tests. Going this way is good since most frameworks are designed to allow you to run any single test, or set of tests that you want to. And if your tests are well designed they can be run singly without having to expect the system was left in some perfect state by a prior test. Thus these sorts of setup/teardown actions are designed to work that way as well.

As an example Cucumber has this at the feature level, with the idea of a 'background' which is basically intended as a way to dry out scenarios by defining common steps to run before each scenario in a feature file. (such as navigating to and logging into your site) This could include a call to a series of steps that would look to see if a browser object existed, and if not create one. However you'd need to put that in every feature file which starts to become rather non dry.

Fortunately cucumber also allows a way to do this in one place via the use of Hooks. You can define hooks to run before steps, in the event of specific conditions, 'before' and 'after' each scenario, as well as code that runs once before any scenarios, and code defined to run 'at_exit' where you could close the browser after all scenarios have run.

If I was using cucumber I'd look at the idea of a some code in env.rb that would run at the start to create a browser, complemented by at_exit code to close the browser. Then perhaps also code in a before hook which could check to see that the browser is still there and re-create it if needed, and maybe logout actions in a after hook. Leave stuff like logging in for the individual scenarios, or a background block if all scenarios in a feature login with the same sort of user.

Hinda

++ for great considerations for when this small piece of my engine grows to warrant its own framework. Not sure I'm there yet, but separating setup & teardown from many sets of tests/actions does seem to be a good fit structurally. Anyway, I noticed discussion on PersistentWebdriver but either it doesn't seem ready for prime time, or is too unclear to me how to apply seamlessly to my current Watir-webdriver browser instance use in Ruby. – Novena

I'd advise to start using an existing framework early. You gain a lot of benefits in terms of unified reporting, setup/teardown and many other wheels you might otherwise spend a lot of time re-inventing. See watir.com/frameworks for a few examples. Then, if you find none of the existing frameworks meet your needs, you can consider either rolling your own, or extending one of the ones that is closest to what you want. A lot of good work has gone into what's out there, it's a shame not to leverage it to make your life a little easier (and address your current question!) – Hinda

These Hooks you pointed out seem a useful feature. Although the current overall design doesn't justify summoning all of Cucumber just for that, because it won't really improve maintainability. The web-driving components are close to final and straightforward already. Very true that retrofitting is always more painful later though. – Novena

Answer 4

Not so much a solution but a workaround for part 1 of my question, using pkill. Posting here since it turned out to be a lot less trivial than I had hoped.

After the ruby script exits, its spawned processes (which may not at all belong in the same PID tree anymore, like firefox-bin) have a predictable "session leader" which turned out to be the parent of the bash shell calling rubyprogram.rb in my case. Available as $PPID in Bash, for when you have to go higher than $$.

Thus to really clean up unwanted heavyweight processes eg. after a ruby crash:

#!/bin/bash
# This is the script that wraps on top of Ruby scripts

./ruby_program_using_watirwebdriver_browser.rb  myparams &  # spawn ruby in background but keep going below:

sleep 11 # give Ruby a chance to launch its web browser

pstree -panu $$  # prints out a process tree starting under Bash, the parent of Ruby. Firefox may not show!

wait  # now wait for Ruby to exit or crash

pkill -s $PPID firefox-bin  # should only kill firefox-bin's caused above, not elsewhere on the system

# Another way without pkill, will also print out what's getting killed if anything:
awk '$7=="firefox-bin" && $3=="'$PPID'" {print $1}' <(ps x -o pid,pgid,sess,ppid,tty,time,comm) | xargs -rt kill

OPTIONAL And since I use a dedicated Xvfb Xwindows server just for webdriving on DISPLAY :99, I can also count on xkill:

timeout 1s  xwininfo -display :99 -root -all |awk '/("Navigator" "Firefox")/ {print $1}' |xargs -rt  xkill -display :99 -id  
# the timeout is in case xkill decides to wait for user action, when window id was missing

Novena

pkill -s $PPID firefox-bin isn't always enough. In other environments (eg. CRON) I had to find who is the session leader of $$ with sesspid=$(awk '$1=="'$$'" {print $3}' <(ps x -o pid,pgid,sess,ppid,tty,time,comm)) and then use that rather than $PPID, as in pkill -s $sesspid firefox-bin – Novena

Answer 5

Just an update on part 2 of my question.

It seems one CAN serialize a Watir:Browser object with YAML, and because it's text-based the contents were quite interesting to me (e.g. some things I've only dreamed of tweaking hidden inside private elements of private classes...but that's a separate topic)

Deserializing from YAML is still trouble. While I haven't tested beyond the first try it gives me some kind of reg exp parse error...not sure what that's about.

(more on that at at how to serialize an object using TCPServer inside? )

Meanwhile, even attempting to serialize with Marshal, which is also built-in to Ruby but stores in binary format, results in a very reasonable-sounding error about not being able to dump a TCPServer object (apparently contained within my Watir:Browser pointed to by $browser)

All in all I'm not surprised at these results, but still pretty confident there is a way, until Watir arrives at something more native (like PersistentWebdriver or how it used to be in the days of jssh when you could simply attach to an already running browser with the right extension)

Until then, if serialization + deserialization to a working object gets too thorny I'll resort to daemonizing a portion of my Ruby to keep objects persistent and spare the frequent and costly setup/teardowns. And I did take a gander at some established (unit testing) frameworks but none seem to fit well yet within my overall software structure--I'm not web testing after all.

Novena

A method exists to allow Webdriver's invocation of Firefox to attach to a running firefox-bin process, by stripping --no-remote : code.google.com/p/selenium/issues/detail?id=2163#c8 – Novena