Parse HTML Infinite Scroll

Question

I'm trying to parse the HTML of a page with infinite scrolling. I want to load all of the content so that I can parse it all. I'm using Python. Any hints?

Wale

Answer 1

Those pages update their html with AJAX. Usually you just need to find the new AJAX requests send by browser, guess the meaning of the AJAX url parameters and fetch the data from the API.

API servers may validate the user agent, referer, cookie, oauth_token ... of the AJAX request, keep an eye on them.

Henderson

Answer 2

the data is

1. either loaded in advance

2. or the page sends a request while you scroll

you can use httpfox to find the request and send it

Brandi