How to code an automated bot that can browse and do operations on a webpage

Question

I need to code a bot that needs to do the following:

Go to a jsp page and search for something by:

• 1: writing something on a search box
• 2: clicking the search button(submit button)
• 3: clicking one of the the resulting buttons/links(same jsp page with different output)
• 4: get the entire html of the new page(same jsp page with different output)

The 4th one can be done with screen scraping and I do not think I need help with it. But I need some guidance to do the options from 1 to 3. Any links or just some keyword that will help me google to learn about it will be appreciated. I plan to do this with java.

Aguayo

Answer 1

Maybe it's not what you want, but you can try selenium : http://seleniumhq.org/

It's a web application testing system.

Tullus

Answer 2

All you need is HTMLUnit

This is an extract from its description

HtmlUnit is a "GUI-Less browser for Java programs". It models HTML documents and provides an API that allows you to invoke pages, fill out forms, click links, etc... just like you do in your "normal" browser.

P.S.: Had used it to build a web scraping project ;)

Chorizo

Oh yeah, HTMLUnit is the tool of choice in that use case, i also used it (through Groovy) for scraping the web. – Loft

and it is worth to mention that it is really annoying that it does not support css selectors but xPath.. -.- – Vyner

Answer 3

const puppeteer = require('puppeteer');

async function searchJSPPage(searchTerm) {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  
  // go to the JSP page
  await page.goto('http://example.com/search.jsp');

  // input search term and submit form
  await page.type('input[name="searchBox"]', searchTerm);
  await page.click('input[type="submit"]');
  await page.waitForNavigation();

  // click on one of the resulting links
  await page.click('a[href="/results/1"]'); // example link
  await page.waitForNavigation();

  // get the entire HTML of the new page
  const html = await page.content();
  console.log(html);

  await browser.close();
}

searchJSPPage("example search term");

Please note that this is just an example, the actual JSP page might have different attributes on the input and button, also the JSP page might have some security mechanisms like CAPTCHAs or cookies that the bot need to handle.

Anhedral

Answer 4

You can use python-mechanize for this.

Overzealous

Answer 5

Prerequistes:

1. Selenium API.
2. Mozilla Firefox (with firebug extension installed)

We can achieve launching of a browser,go to the particular web page ,search a keyword and analyse results by doing following

1. Launch web browser(driver.launch()(selenium)
2. Go to the particular webpage(driver.get("your web pager"))(selenium)
3. Identify the search box(get identifier by using fire bug(id,xml path.. etc)
4. Go to that box and write your search keyword (webelement.sendkeys("your keyword") and click on search button (webelement.click())(selenium)
5. Click on desired result by using identifiers and for next web page to load (selenium)

Apetalous

the answer is not exhaustive. Please provide a sample code that has worked for you – Striptease

Answer 6

I used selenium in chrome. If you want to use selenium you have to download from http://www.seleniumhq.org/download/ --- the latest version and implement in neatbeans or eclipse the jar files. (Selenium Client & WebDriver Language Bindings, Selenium Standalone Server) After this you have to download from google https://sites.google.com/a/chromium.org/chromedriver/ -- chrome driver also the latest version extract the file and save on your pc.

Leathers