Java Runtime.getRuntime().exec() alternatives

Question

I have a collection of webapps that are running under tomcat. Tomcat is configured to have as much as 2 GB of memory using the -Xmx argument.

Many of the webapps need to perform a task that ends up making use of the following code:

Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(command);
process.waitFor();
...

The issue we are having is related to the way that this "child-process" is getting created on Linux (Redhat 4.4 and Centos 5.4).

It's my understanding that an amount of memory equal to the amount tomcat is using needs to be free in the pool of physical (non-swap) system memory initially for this child process to be created. When we don't have enough free physical memory, we are getting this:

    java.io.IOException: error=12, Cannot allocate memory
     at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
     at java.lang.ProcessImpl.start(ProcessImpl.java:65)
     at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
     ... 28 more

My questions are:

1) Is it possible to remove the requirement for an amount of memory equal to the parent process being free in the physical memory? I'm looking for an answer that allows me to specify how much memory the child process gets or to allow java on linux to access swap memory.

2) What are the alternatives to Runtime.getRuntime().exec() if no solution to #1 exists? I could only think of two, neither of which is very desirable. JNI (very un-desirable) or rewriting the program we are calling in java and making it it's own process that the webapp communicates with somehow. There has to be others.

3) Is there another side to this problem that I'm not seeing that could potentially fix it? Lowering the amount of memory used by tomcat is not an option. Increasing the memory on the server is always an option, but seems like more a band-aid.

Servers are running java 6.

EDIT: I should specify that I'm not looking for a tomcat specific fix. This problem can be seen with any of the java applications we have running on the webserver (there are multiple). I simply used tomcat as an example because it will most likely have the most memory allocated to it and it's where we actually saw the error the first time. It is a reproducible error.

EDIT: In the end, we solved this problem by re-writing what the system call was doing in java. I feel that we were pretty lucky being able to do this without making additional system calls. Not all processes will be able to do this, so I would still love to see an actual solution to this.

Heteromorphic

Related: #210375 – Pitchblende

I didn't actually see anything in that article that answered my questions. One action I saw in there was "decrease the amount of memory being used by the parent process" (not an option for us) whether with ulimit or java opts. The other was the same as luke's answer above, which is to make a separate process that uses less memory. This is far from ideal, but at least plausible – Heteromorphic

What JVM are you using (Sun, OpenJDK...?) This doesnt help? #1125271 – Axial

This is the Sun JVM on the mentioned linux OSes. In that article, the original poster says he fixed it with 'echo 0 > /proc/sys/vm/overcommit_memory' and asks someone to explain why. No one ever did. I'm leery of using this method until I have a little more information about it. Google searches didn't help me. This command could possibly answer my #1 or #3, but I'm looking for someone to explain what this is doing and the upsides/downsides of using it. – Heteromorphic

So, what did you do finally?. I think this is an excellent question. Please provide your solution(attempt?) to solve this problem for the benefit of the rest of us. – Skink

I believe this article (though old) discusses this very problem developers.sun.com/solaris/articles/subprocess/subprocess.html – Scalar

Answer 1

I found a workaround in this article, basically the idea is that you create a process early on in the startup of your application that you communicate with (via input streams) and then that subprocess executes your commands for you.

//you would probably want to make this a singleton
public class ProcessHelper
{
    private OutputStreamWriter output;
    public ProcessHelper()
    {
        Runtime runtime = Runtime.getRuntime();
        Process process = runtime.exec("java ProcessHelper");
        output = new OutputStreamWriter(process.getOutputStream());
    }
    public void exec(String command)
    {
        output.write(command, 0, command.length());
    }
}

then you would make a helper java program

public class ProcessHelper
{
    public static void main(String[] args)
    {
         BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
         String command;
         while((command = in.readLine()) != null)
         {
             Runtime runtime = Runtime.getRuntime();
             Process process = runtime.exec(command);
         }
    }
}

what we've essentially done is make a little 'exec' server for your application. If you initialize your ProcessHelper class early on in your application it will successfully create this process then you simply pipe commands over to it, because the second process is much smaller it should always succeed.

You could also make your protocol a little more in depth, such as returning exitcodes, notifying of errors and so on.

Gamecock

This will not work propably for a webapp. The webapp is deployed anytime during the server is running - consuming any amount of memory that time. – Sabba

@Arne Burmeister - that is true, if you are running this on a shared TomCat instance then you might have an arbitrarily large amount of allocated to the process by the time this runs. But if you are running on a nonshared TomCat Server (this is the only application) and you can hook into the application startup to execute this it should work. – Gamecock

As I recently state above, I'm not looking for a tomcat specific answer. We have tomcat with multiple webapps as well as multiple other java processes that could all potentially see the same problem. – Heteromorphic

Answer 2

Try using a ProcessBuilder. The Docs say that's the "preferred" way to start up a sub-process these days. You should also consider using the environment map (docs are in the link) to specify the memory allowances for the new process. I suspect (but don't know for certain) that the reason it needs so much memory is that it is inheriting the settings from the tomcat process. Using the environment map should allow you to override that behavior. However, note that starting up a process is very OS specific, so YMMV.

Philpott

I had looked at ProcessBuilder and specifically it's ability to modify the environment. You can actually do that with Runtime as well. However, I couldn't find any information on exactly what environmental variable I should change to keep the initial memory low. I printed out the current environments of some processes and a memory modifying parameter was not apparent. Can you point out how I could control the memory allocation with environment variables? In Linux specifically? – Heteromorphic

As far as I understand, the ProcessBuilder suffers from the same memory constraints on Linux. – Geneviegenevieve

The reason it needs so much memory is that the standard 'fork' algorithm for spinning off a new process causes the executing process to be entirely duplicated, temporarily. It is said that this is the only cross-platform way to launch sub-processes. As far as I have been able to make out, ProcessBuilder only provides convenience - it doesn't actually resolve this. – Armillary

How exactly does this help? Does ProcessBuilder let you control what system call you'll use to finally launch the new process? That does not seem likely, and hence this does not solve the OP's problem. – Pyrostat

Answer 3

I think this is a unix fork() issue, the memory requirement comes from they way fork() works -- it first clones the child process image (at whatever size it currently is) then replaces the parent image with the child image. I know on Solaris there is some way to control this behavior, but I don't know offhand what it is.

Update: This is already explained in From what Linux kernel/libc version is Java Runtime.exec() safe with regards to memory?

Fourier

I didn't actually see anything in that article that answered my questions. One action I saw in there was "decrease the amount of memory being used by the parent process" (not an option for us) whether with ulimit or java opts. The other was the same as luke's answer above, which is to make a separate process that uses less memory. This is far from ideal, but at least plausible. – Heteromorphic

Answer 4

This would help I think. I know this is an old thread, just for future refs... http://wrapper.tanukisoftware.com/doc/english/child-exec.html

Elviraelvis

Unfortunately, this is part of the Professional Edition, which is expensive, if this is the only thing you ever need. Do you know of any free substitute? – Geneviegenevieve

Try this free alternative to the Tanuki Wrapper: sourceforge.net/projects/yajsw/forums/forum/810311/topic/… – Geneviegenevieve

Answer 5

Another alternative is to have a separate exec process running that watches either a file or some other type of "queue". You append your desired command to that file, and the exec server spots it, running the command, and somehow writing the results to another location you can get.

Isborne

The key being that the exec consumer process has a small footprint, and thus can fork without issues. – Downpipe

Using a file or other publicly visible entry point seems like it could introduce security problems, could it not? Letting an attacker execute arbitrary code as your user (tomcat user in this case). A socket with some sort of authentication protocol might be better. And in practice, that authentication protocol might as well be SSH... – Glassware

Answer 6

Best solution i found so far was using secure shell with public keys. Using http://www.jcraft.com/jsch/ i created a connection to localhost and executed the commands like that. Maybe it has a little more overhead but for my situation it worked like a charm.

Obi