crawl dynamic web page using htmlunit

Question

I am crawling data using HtmlUnit from a dynamic webpage, which uses infinite scrolling to fetch data dynamically, just like facebook's newsfeed. I used the following sentence to simulate the scrolling down event:

webclient.setJavaScriptEnabled(true);
webclient.setAjaxController(new NicelyResynchronizingAjaxController());
ScriptResult sr=myHtmlPage.executeJavaScript("window.scrollBy(0,600)");
webclient.waitForBackgroundJavaScript(10000);
myHtmlPage=(HtmlPage)sr.getNewPage();

But it seems myHtmlPage stays the same with the previous one, i.e., new data is not appended in myHtmlPage, as a result I can only crawl the first few data on the web page. Thanks for your help!

Chemistry

Have you found a way around ? – Munition

Looking for solutions too. Any results? – Effluent

Answer 1

I was searching the same thing. I was only able to find that it is not scroll event (90% sure). There is link on JS wich is responsilbe for loading the page and could maybe help you.

Martial

Answer 2

I had similiar problem where the content were post-loaded during page scrolling. I solved it using:

webClient.getCurrentWindow().setInnerHeight(Integer.MAX_VALUE);

Colpitis