Limiting memory usage of a Java/Rhino/Nashorn object

Question

I'm extending a server application written in Java to allow user-defined callbacks (written in Javascript) to be run in response to requests. I've done some reading, and while it seems possible to disable Java classes in Nashorn, there is nothing stopping a user from creating Javascript code that allocates an enormous array without using any Java APIs. I'm wondering if there is any way to restrict this, either proactively or reactively.

The solution I came up with is to have a process pool of JVMs with small max heap sizes, which are responsible for running the user-defined code. There will be a worker pool manager to spawn new processes when needed. This way, the main process, as well as other user-defined code, will not be affected by a single malicious user. While this solution will likely work, it seems heavy-handed. Is there no better solution for preventing malicious users from using too much memory?

I'm not particularly set on Javascript, so if there exists any other scripting language that can be run within a JVM and also has support for memory usage limits, I would be open to using it instead of Nashorn. Unfortunately, it seems like Jython, JRuby, and LuaJava all don't have what I'm looking for. Thanks in advance.

Glyphography

I wonder if the newArray() and newObject() hooks on Context could be used to track memory usage. You could then kill it on those calls or in observeInstructionCount() – Determinative

Interesting idea. How could I extend this to general memory usage, say if someone creates an array, and then grows it using push? – Glyphography

This is where it gets ugly. The objects you return need to be decorated/subclassed so they monitor those operations. The problem is grow and slots are private in ScriptableObject (slots is the hashtable that backs objects), and there isn't a good API for getting the slot count. You'd also have to only return objects that are registered with a ReferenceQueue so that you can track them when they're GCed. – Determinative

Thanks for your input. Given everything you've said, I'm not confident that such a tailored solution wouldn't have a hole somewhere, making the restrictions for naught. Although using separate processes has more overhead, it would guarantee memory limits that could not be circumvented. It also has the additional benefit of being generalizable to any scripting language that has a JVM-based interpreter. – Glyphography