Is it possible to use a bot to open a browser, manually manipulate the page, then continue using a bot on it?

Question

I'm using Ruby, Selenium WebDriver and Nokogiri to retrieve data from webpages. Once the proper HTML is loaded, I print the contents of a certain class.

For example,

require "selenium-webdriver"
require "nokogiri"
browser = Selenium::WebDriver.for :chrome
browser.get "https://jsfiddle.net"
doc = Nokogiri::HTML.parse(browser.page_source)
doc.css('.aiButton').map(&:text).join(',')

I've found by far the hardest part is getting the correct HTML loaded properly. For example, the content I want might be hidden by some javascript, or might be on different page.

Is it possible to use Selenium to load the page, then manually manipulate the page so the correct HTML is displayed, and then allow the bot to finish and print the content it's supposed to?

Cordate

the short answer is yes. Selenium will open a browser in your GUI and it will remain open until your script is done. – Craal

Answer 1

You can use Selenium to interact with the webpage - fill form fields, click buttons etc. You can even execute your own javascript code.

Selenium cheat sheet

Edit:

Using pry to stop the code execution so you can manually manipulate the web page.

# Code for starting Selenium session and opening the web page
...

# Use pry to stop the code execution.
# Resume the program using command 'exit' in the pry context
require 'pry'; binding.pry

# Code to get results after you manually manipulate the web page
...

Tolbert

I've found that's the most time-consuming part. Is there a way of doing that part manually and then resuming the bot after I'm done? – Cordate

I don't know why you would do that. Selenium driver will always interact with the webpage faster than you would be able to manually. – Tolbert

It's because every use case will be different, so I would have to write different code every time. It's quicker to just get the HTML situated myself. – Cordate

I don't know if you can continue session you started manually with Selenium. I think it would be more complicated than modifying script for retrieving data. If you write the code in reusable way, it will not be such overhead. – Tolbert

You can put a breakpoint in your code when you want the script to pause, run it, manipulate the page, and then continue the script run... but I'm with kallax, I don't know why you would want to do that. – Nadabb

@Nadabb How can I pause and then resume the script? – Cordate

It depends on the IDE that you use. I would read the docs for your IDE and see how to use breakpoints. – Nadabb

Answer 2

You can do this quite easily. I am not familiar with ruby, but I'll outline the process.

1) start driver 2) go to your page 3) then ask for user input (in python 2 eg: continue = raw_input('type something and hit enter here in the console to continue'))

4) then do all the other things you want to do.

When you execute this script it will stop at the question. Then you can manually manipulate the browser and when you're done you go to the console/cmd window and type "go" and hit enter. Then it will continue from where you manually left the browser.

Coaxial