I'm working on Scrapy to crawl pages,however,I can't handle the pages with javascript. People suggest me to use htmlunit, so I got it installed,but I don't know how to use it at all.Dose anyone can...

I cannot, for the life of me, rig HtmlUnit up to grab this site: http://www.bing.com/travel/flight/flightSearch?form=FORMTRVLGENERIC&q=flights+from+SLC+to+BKK+leave+07%2F30%2F2010+return+08%2F...

According to https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot, with HtmlUnit (2.13) I am trying to create a snapshot for a webpage using AngularJS (1.2.1). My Java code i...

I'm looking to gather information from a set of web pages that are all very similarly formatted. I need some information that is loaded onto the page by Javascript after opening. It seems that HTML...

My project includes htmlunit jars and downloads some pages content. Executable jar (which includes libs, funct. of eclipse export) thereof, however, works only on the machine on which I created it ...

Copied from here: I need to save files from websites using HtmlUnit. I am currently navigating to pages that have several anchors that use javascript: onClick()="DownloadAttachment('attachmentId...

I am using HtmlUnitDriver,& here is my code. HtmlUnitDriver driver = new HtmlUnitDriver(true); driver.get("some url here"); I am getting following Exception: Caused by: com.gargoylesoftwa...

I'm trying to call the function showPage('3'); of this page, for use the page source code after. I tried to do with htmlUnit like so: WebClient webClient = new WebClient(); webClient.waitForBack...

Is it possible to teach HTMLUnit to ignore certain javascript scripts/files on a web page? Some of them are just out of my control (like jQuery) and I can't do anything with them. Warnings are anno...

I'm trying to click on a search button in this website: http://www.amadeusepower.com/trek/portals/trek/default.aspx?Culture=en-US The button is in here somewhere <table cellpadding="0" cells...

I experienced slowness in HTMLUnit 2.12, and therefore disabled CSS as explained in HTMLUnit : super slow execution?. I want to understand what the trade-off is. Does it mean that I cannot use XPa...

Here is my code to get the page: WebClient webClient = new WebClient(); HtmlPage page = webClient.getPage(url); The problem is the webClient always executes javascript automatically and throws m...

I was trying to use http unit to read response header for my application - WebClient webClient = new WebClient(); WebClient.setThrowExceptionOnScriptError(false); HtmlPage currentPage = webClient....

I have been using HTMLUnit . It suits my requirements well. But it seems to be extremely slow. for example : I have automated the following scenario using HTMLUnit Goto Google page Enter some text...

I'm having some trouble figuring out how to get the content of some HTML after javascript has updated it. Specifically, I'm trying to get the current time from US Naval Observatory Master Clock. It...

I'm playing with a grails app that has a contextmenu (on right-click). The context menu is built using Chris Domigan's jquery contextmenu plugin. While the contextmenus do actually work, I want to...

Has anybody gotten HTMLUnit (or HtmlUnitDriver) to work on Android apps? This is the problem: I am getting the following error message: 11-26 16:27:26.617: E/AndroidRuntime(1265): java.lang.NoCl...

I have developed a script whose sole purpose is to check if the website/service is up and running. The way it does that is ,it connects to the page using its url and logs in to the site using the u...

I'm new to maven and jasmine, and I have a problem setting up maven to run unit tests on an angular application. I get into trouble as soon as I add angular-mocks.js to the test/javascript/lib fold...

I'm trying out HTMLUnit to automate downloading data off a webapp. However, I am getting a whole mess of warnings on getPage() (most of which seem to deal with linked scripts that I don't think i e...

I am trying to get page source of following URL using Html-Unit get method. http://denydesigns.com/collections/barbara-sherman-fleece-throw-blanket/products/barbara-sherman-antique-fleece-throw-b...

I'd like to save HtmlUnit cookies to a file and on next run load them from that one. How can I do that? Thanks.

I am trying to be able to test a website that uses javascript to render most of the HTML. With the HTMLUNIT browser how would you be able to access the html generated by the javascript? I was looki...

I am building an application in C# which uses com.gargoylesoftware.htmlunit.WebClient to access and retrieve information from webpages. My application runs fine from the main project but when I tr...

Does Python have screen scraping libraries that offer JavaScript support? I've been using pycurl for simple HTML requests, and Java's HtmlUnit for more complicated requests requiring JavaScript su...