Headless Browser for Python (Javascript support REQUIRED!) [closed]

G

6

57

I need a headless browser which is fairly easy to use (I am still fairly new to Python and programming in general) which will allow me to navigate to a page, log into a form that requires Javascript, and then scrape the resulting web page by searching for results matching certain criteria, clicking check boxes, and clicking to download files. All of this requires Javascript.

I hear a headless browser is what I want - requirements/preferences are that I be able to run it from Python, and preferably that the resultant script will be compilable by py2exe (I am writing this program for other users).

So far Windmill looks like it MIGHT be what I want, but I am not sure.

Any ideas appreciated!

Gusset answered 17/5, 2011 at 0:45 Comment(2)

Sorry, as far as I know this does not exist (yet). The best you can do now is run webdriver, driven from the Python interface. You can drive HtmlUnit that way, but that is written in Java so you have a combination of Java and Python. – Leicestershire 17/5, 2011 at 3:43

Possibly related: #13287990 – Dick 10/6, 2013 at 9:35

G

11

The answer to this question was Spynner

Gusset answered 9/11, 2011 at 18:21 Comment(3)

Spynner's dependency Libxslt requires vcvarsall.bat from VS 2008 which creates quite an ordeal seen here: link We need an alternative. – Aguste 11/1, 2014 at 20:33

Looks like Spynner does not support python3 ATM – Milla 24/1, 2018 at 3:47

I'm not sure if this is the best answer at the present time, but in 2011 Python 2 was very viable – Gusset 24/1, 2018 at 3:49

D

30

I use webkit as a headless browser in Python via pyqt / pyside:
http://www.riverbankcomputing.co.uk/software/pyqt/download
http://developer.qt.nokia.com/wiki/Category:LanguageBindings::PySide::Downloads

I particularly like webkit because it is simple to setup. For Ubuntu you just use: sudo apt-get install python-qt4

Here is an example script:
http://webscraping.com/blog/Scraping-JavaScript-webpages-with-webkit/

Diena answered 17/5, 2011 at 4:11 Comment(5)

There's also PySide, which is similar to PyQt except under LGPL instead of GPL. – Hurds 17/5, 2011 at 4:14

I think headless would imply no actual browser page, which while you can do that with webkit - I have found it to be useful being driven by Python. The only problem is that interacting with Javascript is not the easiest thing in the world - if I remember correctly you can't just inject stuff. But, I did manage to get a Python interpeter embedded into a Qt App that could 'drive' the webkit interface, so it's definitely got some juice. You might also want to take a look at sikuli.org for more of a test oriented solution. – Calenture 10/6, 2011 at 5:10

This is exactly what I've done for a project I'm working on using Django to have a web interface as well as a cross-platform qt interface. this way I can have feature parity at a very low cost. – Villosity 15/6, 2011 at 8:13

@synthesizerpatel: webkit can be run headless and you can inject javascript via frame.evaluateJavaScript() – Diena 20/12, 2011 at 8:28

Any future visitors may wish to check out Ghost.py which provides a nice wrapper around PyQt/PySide. – Overstudy 27/4, 2012 at 3:27

G

11

The answer to this question was Spynner

Gusset answered 9/11, 2011 at 18:21 Comment(3)

Spynner's dependency Libxslt requires vcvarsall.bat from VS 2008 which creates quite an ordeal seen here: link We need an alternative. – Aguste 11/1, 2014 at 20:33

Looks like Spynner does not support python3 ATM – Milla 24/1, 2018 at 3:47

I'm not sure if this is the best answer at the present time, but in 2011 Python 2 was very viable – Gusset 24/1, 2018 at 3:49

B

9

I'm in the midst of writing a Python driver for Zombie.js, "a lightweight framework for testing client-side JavaScript code in a simulated environment".

I'm currently at a standstill on a resolution to a bug in Node.js (before I write more tests and more code), but feel free to keep an eye on my project as it progresses:

https://github.com/ryanpetrello/python-zombie

Binkley answered 9/6, 2011 at 4:0 Comment(1)

Are you guys still maintaining it ? Just checked, build is failing. – Co 31/10, 2014 at 2:21

H

5

There are not too many headless browsers yet that support Javascript.

You could try Zombie.js or Phantomjs. Those are not Python, but plain Javascript and those really can do the job.

Hiltner answered 28/5, 2011 at 0:57 Comment(1)

Related: #13287990 – Dick 10/6, 2013 at 9:36

G

3

Try using phantomjs, it has great javascript support. Then you could run it as a subprocess of a python script

http://docs.python.org/library/subprocess.html

that could boss it around.

Ghostly answered 11/6, 2011 at 2:51 Comment(1)

I do this, looking to replace it as it's quite intensive to stop/start the process which greatly impacts performance. Then, once you start running phantomjs as a service you will encounter a range of issues like memory leaks. – Cotta 13/3, 2016 at 22:24

P

1

You can use HTQL in combination with IRobotSoft webscraper. Check here for examples: http://htql.net/

Palish answered 28/5, 2011 at 0:51 Comment(0)

Recommended topics

Hot tags