Preventing System.exit() from API

Question

I am using a third party library that does a System.exit() if it encounters exceptions. I am using the APIs from a jar. Is there anyway that I can prevent the System.exit() call because it causes my application to shutdown? I cannot decompile and recompile the jar after removing the System.exit() because of a lot of other licensing issues. I once came across an answer [to some other question that I do not remember] in stackoverflow that we can use the SecurityManager in Java to do something like this.

Windom

IMO, a third party library that calls System.exit() is broken, by definition. If they won't fix this, I'd consider looking for a better quality alternative. – Addressee

May i know the library that you are using? – Thresher

@Suresh...it is a library owned by the company I am working for. It is against company policy to reveal the name. – Windom

Another reason why I hate closed source libraries... – Hectoliter

Answer 1

You can install a security manager which disables System.exit():

  private static class ExitTrappedException extends SecurityException { }

  private static void forbidSystemExitCall() {
    final SecurityManager securityManager = new SecurityManager() {
      public void checkPermission( Permission permission ) {
        if( "exitVM".equals( permission.getName() ) ) {
          throw new ExitTrappedException() ;
        }
      }
    } ;
    System.setSecurityManager( securityManager ) ;
  }

  private static void enableSystemExitCall() {
    System.setSecurityManager( null ) ;
  }

---

Edit: Max points out in comments below that

as of Java 6, the permission name is actually "exitVM."+status, e.g. "exitVM.0".

However, the permission exitVM.* refers to all exit statuses, and exitVM is retained as a shorthand for exitVM.*, so the above code still works (see the documentation for RuntimePermission).

Heartbreak

+1 - This is the way. It means that the 3rd party library will trigger a SecurityException if it calls System.exit() ... so your code has to cope with this. – Addressee

System.setSecurityManager( null ) ; Seems a very hackish approach to me. The developer has just opened the JRE for any code to do anything. While the extended SecurityManager I outlined was also a toy version, it at least has the potential to provide further control over 3rd party APIs. It also seems that approach might fail for 3rd party APIs that start their own threads. – Deplorable

As good as it may be, this answer will become utterly worthless as soon as the link breaks... It is generally encouraged to at least summarize the main points of the answer rather than just post a link. – Department

Update, as of Java 6, the permission name is actually "exitVM."+status, e.g. "exitVM.0". – Dwarfism

There's a couple of issues with the above code. 1) Despite the reassurances on "exitVM."+status and exitVM.*, and the equivalence of exitVM in Java 6+, the code above will not work as-is, because the permission-wildcarding is applied at a lower level than the overridden method. permission.getName() of exitVM.0 (or .1, .2 etc.) will be passed, which won't meet "exitVM".equals. Better is permission.getName().startsWith("exitVM"). 2) If the permission you want to trap is System.exit(), eee java.lang.SecurityManager.checkExit(int) and just override this method instead. – Aircool

The above code sample was very helpful is resolving the problem, but it had the side effect of blocking my code's access to files. I've added my version of the security manager in a separate post. – Kigali

jroller.com/ethdsy/entry/disabling_system_exit seems to be a broken link. Also can't find it on the wayback machine. If anyone knows of a cached version somewhere, I'd love to read it. – Nader

Answer 2

See my reply to How to avoid JFrame EXIT_ON_CLOSE operation to exit the entire application?.

Edit 1: The source that was linked. Demonstrates how to use a SecurityManager to prevent System.exit(n).

import java.awt.*;
import java.awt.event.*;
import java.security.Permission;

/** NoExit demonstrates how to prevent 'child'
applications from ending the VM with a call
to System.exit(0).
@author Andrew Thompson */
public class NoExit extends Frame implements ActionListener {

  Button frameLaunch = new Button("Frame"),
     exitLaunch = new Button("Exit");

  /** Stores a reference to the original security manager. */
  ExitManager sm;

  public NoExit() {
     super("Launcher Application");

     sm = new ExitManager( System.getSecurityManager() );
     System.setSecurityManager(sm);

     setLayout(new GridLayout(0,1));

     frameLaunch.addActionListener(this);
     exitLaunch.addActionListener(this);

     add( frameLaunch );
     add( exitLaunch );

     pack();
     setSize( getPreferredSize() );
  }

  public void actionPerformed(ActionEvent ae) {
     if ( ae.getSource()==frameLaunch ) {
        TargetFrame tf = new TargetFrame();
     } else {
        // change back to the standard SM that allows exit.
        System.setSecurityManager(
           sm.getOriginalSecurityManager() );
        // exit the VM when *we* want
        System.exit(0);
     }
  }

  public static void main(String[] args) {
     NoExit ne = new NoExit();
     ne.setVisible(true);
  }
}

/** This example frame attempts to System.exit(0)
on closing, we must prevent it from doing so. */
class TargetFrame extends Frame {
  static int x=0, y=0;

  TargetFrame() {
     super("Close Me!");
     add(new Label("Hi!"));

     addWindowListener( new WindowAdapter() {
        public void windowClosing(WindowEvent we) {
           System.out.println("Bye!");
           System.exit(0);
        }
     });

     pack();
     setSize( getPreferredSize() );
     setLocation(++x*10,++y*10);
     setVisible(true);
  }
}

/** Our custom ExitManager does not allow the VM
to exit, but does allow itself to be replaced by
the original security manager.
@author Andrew Thompson */
class ExitManager extends SecurityManager {

  SecurityManager original;

  ExitManager(SecurityManager original) {
     this.original = original;
  }

  /** Deny permission to exit the VM. */
   public void checkExit(int status) {
     throw( new SecurityException() );
  }

  /** Allow this security manager to be replaced,
  if fact, allow pretty much everything. */
  public void checkPermission(Permission perm) {
  }

  public SecurityManager getOriginalSecurityManager() {
     return original;
  }
}

Deplorable

I don't think this question is in regards to Java Swing. – Nellynelms

While my reply was based around a GUI, the technique of using a SecurityManager would work for a headless application (perhaps with a few tweaks to the post's source code) just as well as an app. with a GUI. I will edit my post to include the source that some people on this thread apparently ignored. O_o – Deplorable

Should checkPermission delegate checkPermission to original so that TargetFrame does not get more privileges than other code? – Adolfoadolph

@MikeSamuel Good question (which maybe you should ask as a dedicated question). – Deplorable

Answer 3

The previous code sample is partially correct, but I found that it ended up blocking my code's access to files. To get around that problem, I wrote my SecurityManager a little differently:

public class MySecurityManager extends SecurityManager {

private SecurityManager baseSecurityManager;

public MySecurityManager(SecurityManager baseSecurityManager) {
    this.baseSecurityManager = baseSecurityManager;
}

@Override
public void checkPermission(Permission permission) {
    if (permission.getName().startsWith("exitVM")) {
        throw new SecurityException("System exit not allowed");
    }
    if (baseSecurityManager != null) {
        baseSecurityManager.checkPermission(permission);
    } else {
        return;
    }
}

}

In my case, I needed to prevent a 3rd party library from terminating the VM. But there were also some grails tests that were calling System.exit. So, I wrote my code so that it only activated the custom security manager immediately before the call to the 3rd party library (not a common event) and then immediately restored the original security manager, if any, afterwards.

It's all a little ugly. Ideally, I would have preferred to simply remove the System.exit code, but I do not have access to the 3rd party library's source code.

Kigali

Answer 4

Using SecurityManager to disallow System.exit() calls is not perfect, for at least 2 reasons:

1. Java applications running with and without SecurityManager enabled are very different. That's why it needs to be turned off eventually, but it cannot be done with System.setSecurityManager(null). This call will lead to another security permission check, that will inevitably fail, because application code (SecurityManager subclass) is on the top of the calling stack.

2. All the Java applications are multi-threaded, and other threads can do various things between forbidSystemExitCall() and enableSystemExitCall(). Some of these things can be protected with security permission checks, which will fail for the very same reasons as described above. If checkExit() is overridden instead of [much more generic] checkPermission(), it will cover most of the cases, though.

The only way (that I know) to resolve this is to grant all the privileged to the SecurityManager subclass. It will very likely require it to be loaded by a separate class loader, e.g. bootstrap (null) class loader.

Glyndaglynias