NOTE: The answer below is only valid for Java EE 5. As was brought to my attention in one of the other answers, Java EE 6 does support this. So if you're using Java EE 6, don't read this answer, but read the other relevant answer.
From my own research and from the responses to this question the answer I found is the following: Although JAAS is a standard interface, there is no uniform way to write, deploy and integrate a JAAS Realm + LoginModule in various application servers.
Glassfish v2 requires you to extend some of its own internal classes that implement LoginModule or Realm themselves. You can however not customize the whole login process because many methods of the LoginModule interface are marked final in Glassfish's superclass. The custom LoginModule and Realm classes must be placed in the AS classpath (not the application's), and the realm must be manually registered (no deployment from the .war possible).
The situation seems to be a bit better for Tomcat, which will let you code your own Realm and LoginModule completely, and then configure them into the application server using its own JAASRealm (which will delegate the actual work to your implementations of Realm and LoginModule). However, even tomcat doesn't allow deploying your custom realm from your .war.
Note that none of the Application Servers that showed up my results seem to be able to take full advantage of all the JAAS Callbacks. All of them seem to only support the basic username+password scheme. If you need anything more complicated than that, then you will need to find a solution that is not managed by your Java EE container.
For reference, and because it was asked for in the comments to my question, here is the code I wrote for GlassfishV2.
First of all, here's the Realm implementation:
public class WebserviceRealm extends AppservRealm {
private static final Logger log = Logger.getLogger(WebserviceRealm.class.getName());
private String jaasCtxName;
private String hostName;
private int port;
private String uri;
@Override
protected void init(Properties props) throws BadRealmException, NoSuchRealmException {
_logger.info("My Webservice Realm : init()");
// read the configuration properties from the user-supplied properties,
// use reasonable default values if not present
this.jaasCtxName = props.getProperty("jaas-context", "myWebserviceRealm");
this.hostName = props.getProperty("hostName", "localhost");
this.uri = props.getProperty("uri", "/myws/EPS");
this.port = 8181;
String configPort = props.getProperty("port");
if(configPort != null){
try{
this.port = Integer.parseInt(configPort);
}catch(NumberFormatException nfe){
log.warning("Illegal port number: " + configPort + ", using default port (8181) instead");
}
}
}
@Override
public String getJAASContext() {
return jaasCtxName;
}
public Enumeration getGroupNames(String string) throws InvalidOperationException, NoSuchUserException {
List groupNames = new LinkedList();
return (Enumeration) groupNames;
}
public String getAuthType() {
return "My Webservice Realm";
}
public String getHostName() {
return hostName;
}
public int getPort() {
return port;
}
public String getUri() {
return uri;
}
}
And then the LoginModule implementation:
public class WebserviceLoginModule extends AppservPasswordLoginModule {
// all variables starting with _ are supplied by the superclass, and must be filled
// in appropriately
@Override
protected void authenticateUser() throws LoginException {
if (_username == null || _password == null) {
throw new LoginException("username and password cannot be null");
}
String[] groups = this.getWebserviceClient().login(_username, _password);
// must be called as last operation of the login method
this.commitUserAuthentication(groups);
}
@Override
public boolean commit() throws LoginException {
if (!_succeeded) {
return false;
}
// fetch some more information through the webservice...
return super.commit();
}
private WebserviceClient getWebserviceClient(){
return theWebserviceClient;
}
}
finally, in the Realm has to be tied to the LoginModule. This is done at the JAAS configuration file level, which in glassfish v2 lies at yourDomain/config/login.conf. Add the following lines at the end of that file:
myWebserviceRealm { // use whatever String is returned from you realm's getJAASContext() method
my.auth.login.WebserviceLoginModule required;
};
This is what got things working for me on glassfish. Again, this solution is not portable across application servers, but as far as I can tell, there is no existing portable solution.