Executing Javascript from Python

Question

I have HTML webpages that I am crawling using xpath. The etree.tostring of a certain node gives me this string:

<script>
<!--
function escramble_758(){
  var a,b,c
  a='+1 '
  b='84-'
  a+='425-'
  b+='7450'
  c='9'
  document.write(a+c+b)
}
escramble_758()
//-->
</script>

I just need the output of escramble_758(). I can write a regex to figure out the whole thing, but I want my code to remain tidy. What is the best alternative?

I am zipping through the following libraries, but I didnt see an exact solution. Most of them are trying to emulate browser, making things snail slow.

• http://code.google.com/p/python-spidermonkey/ (clearly says it's not yet possible to call a function defined in Javascript)
• http://code.google.com/p/webscraping/ (don't see anything for Javascript, I may be wrong)
• http://pypi.python.org/pypi/selenium (Emulating browser)

Edit: An example will be great.. (barebones will do)

Toratorah

Wait. Is this a dupe? Or was Pyjamas wrong enough that somebody deleted their answer? – Forced

May be PhantomJS can help or pyv8. – Fulgurant

@ErikReppen I checked Pyjamas, it has no examples. – Toratorah

@Fulgurant request you to show an example if possible. – Toratorah

I deleted it because I realised it was pretty wrong. – Underskirt

Then again you don't really need SpiderMonkey or V8 to compute by hand what escramble() does :-) – Sloatman

Answer 1

Using PyV8, I can do this. However, I have to replace document.write with return because there's no DOM and therefore no document.

import PyV8
ctx = PyV8.JSContext()
ctx.enter()

js = """
function escramble_758(){
var a,b,c
a='+1 '
b='84-'
a+='425-'
b+='7450'
c='9'
document.write(a+c+b)
}
escramble_758()
"""

print ctx.eval(js.replace("document.write", "return "))

Or you could create a mock document object

class MockDocument(object):

    def __init__(self):
        self.value = ''

    def write(self, *args):
        self.value += ''.join(str(i) for i in args)


class Global(PyV8.JSClass):
    def __init__(self):
        self.document = MockDocument()

scope = Global()
ctx = PyV8.JSContext(scope)
ctx.enter()
ctx.eval(js)
print scope.document.value

Jessejessee

How do I install PyV8? When I do a python setup.py install, I get ERROR: you should set V8_HOME to the Google v8 folder, or download and build it first. <code.google.com/p/v8> On visiting this project I just see 2 downloads, benchmarks-v2.zip benchmarks-v1.zip. None of them have any py files. – Toratorah

I just use the exe installer because I'm using Window. I'm not sure about how to install it on other platforms. – Jessejessee

I was facing the same problem and wow this is great. It works perfectly. Now I'm wondering how can I manage the function prompt?! help plz. – Brutality

PyV8 is really really broken! First of all, it requires a very specific version of Python and V8. Second, you need to guess for yourself. – Delanadelancey

Do you know of any other JS interpeters for Python that allow the creation of mock objects that can be implemented as a scope? I've been looking at PyMiniRacer but haven't come across anything for adding a python class into the scope? Thx – Andel

replit.com/@RixTheTyrunt/EaseJS?v=1 Barely doing the job – Laurettalaurette

@Laurettalaurette there is nothing on replit.com. Just an empty replit... – Blades

Answer 2

You can also use Js2Py which is written in pure python and is able to both execute and translate javascript to python. Supports virtually whole JavaScript even labels, getters, setters and other rarely used features.

import js2py

js = """
function escramble_758(){
var a,b,c
a='+1 '
b='84-'
a+='425-'
b+='7450'
c='9'
document.write(a+c+b)
}
escramble_758()
""".replace("document.write", "return ")

result = js2py.eval_js(js)  # executing JavaScript and converting the result to python string 

Advantages of Js2Py include portability and extremely easy integration with python (since basically JavaScript is being translated to python).

To install:

pip install js2py

Followthrough

Do not use this if you're running untrusted code (a scraped HTML page), js2py has a pyimport statement that let's you run arbitrary code. – Substantive

Answer 3

One more solution as PyV8 seems to be unmaintained and dependent on the old version of libv8.

PyMiniRacer It's a wrapper around the v8 engine and it works with the new version and is actively maintained.

pip install py-mini-racer

from py_mini_racer import py_mini_racer
ctx = py_mini_racer.MiniRacer()
ctx.eval("""
function escramble_758(){
    var a,b,c
    a='+1 '
    b='84-'
    a+='425-'
    b+='7450'
    c='9'
    return a+c+b;
}
""")
ctx.call("escramble_758")

And yes, you have to replace document.write with return as others suggested

Alvardo

Hi, do you know whether it is possible to extent the scope of PyMiniRacer to use python classes, like the PyV8 example above? Basically being able to access certain Python classes/objects with the JS? Thanks. – Andel

@Andel nope, never used it for anything but simple js snippets – Alvardo

Answer 4

You can use js2py context to execute your js code and get output from document.write with mock document object:

import js2py

js = """
var output;
document = {
    write: function(value){
        output = value;
    }
}
""" + your_script

context = js2py.EvalJs()
context.execute(js)
print(context.output)

Erythema

Answer 5

You can use requests-html which will download and use chromium underneath.

from requests_html import HTML

html = HTML(html="<a href='http://www.example.com/'>")

script = """
function escramble_758(){
    var a,b,c
    a='+1 '
    b='84-'
    a+='425-'
    b+='7450'
    c='9'
    return a+c+b;
}
"""

val = html.render(script=script, reload=False)
print(val)
# [phone]

More on this read here

Paragraphia

Answer 6

quickjs should be the best option after quickjs come out. Just pip install quickjs and you are ready to go.

modify based on the example on README.

from quickjs import Function

js = """
function escramble_758(){
var a,b,c
a='+1 '
b='84-'
a+='425-'
b+='7450'
c='9'
document.write(a+c+b)
escramble_758()
}
"""

escramble_758 = Function('escramble_758', js.replace("document.write", "return "))

print(escramble_758())

https://github.com/PetterS/quickjs

Mohican

Answer 7

Really late to the party but you can use a successor of pyv8 which is regularly maintained by a reputable organization (Subjective) named CloudFlare. Here is the repository URL:

https://github.com/cloudflare/stpyv8

Bentlee

Answer 8

PythonMonkey is a new alternative that uses Firefox's JS engine.

Just pip install pythonmonkey to get started.

import pythonmonkey as pm

some_js_code = """
function escramble_758() {
  var a,b,c
  a='+1 '
  b='84-'
  a+='425-'
  b+='7450'
  c='9'
  return a+c+b;
}
escramble_758()
"""

res = pm.eval(some_js_code)

print(res) # [phone]

---

You can also directly require JavaScript files from Python using pythonmonkey.require

Selector