Python Mechanize + GAEpython code

Question

I am aware of previous questions regarding mechanize + Google App Engine, What pure Python library should I use to scrape a website? and Mechanize and Google App Engine.

Also there is some code here, which I cannot get to work on app engine, throwing

File “D:\data\eclipse-php\testpy4\src\mechanize\_http.py”, line 43, in socket._fileobject(”fake socket”, close=True)
File “C:\Program Files (x86)\Google\google_appengine\google\appengine\dist\socket.py”, line 42, in _fileobject
fp.fileno = lambda: None
AttributeError: ’str’ object has no attribute ‘fileno’
INFO 2009-12-14 09:37:50,405 dev_appserver.py:3178] “GET / HTTP/1.1″ 500 -

Is anybody willing to share their working mechanize + appengine code?

Sherilyn

Answer 1

I have solved this problem, just change the code of mechanize._http.py, about line 43, from:

try:
    socket._fileobject("fake socket", close=True)
except TypeError:
    # python <= 2.4
    create_readline_wrapper = socket._fileobject
else:
    def create_readline_wrapper(fh):
        return socket._fileobject(fh, close=True)

to:

try:
    # fixed start -- fixed for gae
    class x:
        pass

    # the x should be an object, not a string,
    # This is the key
    socket._fileobject(x, close=True)
    # fixed ended
except TypeError:
    # python <= 2.4
    create_readline_wrapper = socket._fileobject
else:
    def create_readline_wrapper(fh):
        return socket._fileobject(fh, close=True)

Impish

this currently needs to go in _urllib2_fork.py – Nagaland

If you try to do it as of now (Dec 2011) it is not in the same file but in urllib2 – Eada

Answer 2

I managed to get mechanize code that runs on GAE, many thanks to MStodd, from GAEMechanize project http://code.google.com/p/gaemechanize/ and

If anybody needs the code, you can contact MStodd !

ps: the code is not on google code, so you have to contact the owner..

Cheers don

Sherilyn

Answer 3

I've uploaded the source of the gaemechanize project to a new project: http://code.google.com/p/gaemechanize2/

Insert usual caveats.

Slob