How to leave client waiting for Java JAX-RS service to prevent DOS

Question

I'm having an issue with a web service with users trying to guess application IDs by looping over random IDs.

The bad requests are coming from random IPs, so I cannot just ban their IP (unless I do it dynamically, but I'm not looking into that yet).

Currently when I detect a client that has made 10 bad app ID attempts I put them on a block list in my app, and reject further requests from that IP for the day.

I want to minimize the amount of work my server needs to do, as the bad client will continue to send 1000s of requests even though they get rejected. I know there are dynamic Firewall solutions, but want something easy to implement in my app for now. Currently I am sleeping for 5 seconds to reduce the calls, but what I want to do is just not send a response to the client, so it has to timeout.

Anyone know how to do this in Java, in JAX-RS?

My service is like,

@Path("/api")
public class MyServer {

@GET
@Consumes(MediaType.APPLICATION_XML)
@Produces(MediaType.APPLICATION_XML)
@Path("/my-request")
public String myRequest(String type,
    @Context HttpServletRequest requestContext,
    @Context HttpServletResponse response) {
...
}

See: How to stop hack/DOS attack on web API

Foxglove

The attacker probably doesn't care if the request times out. Sounds like it's coming from a bot farm, meaning leaving the connection open forever is irrelevant. I don't know about JAX RS, but in Java I would look at asynchronous servlet (or continuations) to not tie up a server thread sleeping for 5 seconds. Sorry, I don't know how to do that in JAXRS! – Davisson

Adding the sleep reduces the calls, so letting it timeout should reduce calls further, I think it is coming from an Androd app, not a farm – Foxglove

Hey @Foxglove have you tried redirecting the clients to somewhere really slow or a huge file somewhere else? If the client accepts the redirect it would be occupied for a while and you don't need a sleeping thread anymore. – Stpierre

@Stpierre I think the attacker will very soon come with a solution to avoid following redirects :) Also the attacker did not notice that he waits for 5 seconds when blocked, but when he will, he will definitely stop waiting for a response, i.e he will try making parallel requests. So, James, your solution probably won't help for long, except requests really come from Mobile devices, in which case you can hope the owners of mobile devices will notice the eventual big traffic and will uninstall the app. – Agrippina

Can you change/modify your REST API? – Baucom

@AndreiI i'm aware of that and i was going to say that the attack will probobly get more sophisticated. But then i realized he was asking for a quick fix. – Stpierre

Answer 1

You are looking for asynchronous responses which are supported by JAX-RS. The tutorial for Jersey contains some examples of how to implement an asynchronous response to a request.

With asynchronous responses, the thread that is responsible for answering a request is freed for handling another request already before a request is completed. This feature is activated by adding a parameter with the @Suspended annotation. What you would need to do additionally, would be to register a dedicated scheduler that is responsible for waking up your requests after a given time-out like in this example:

@Path("/api")
public class MyServer {

  private ScheduledExecutorService scheduler = ...;

  @GET
  @Consumes(MediaType.APPLICATION_XML)
  @Produces(MediaType.APPLICATION_XML)
  @Path("/my-request")
  public String myRequest(String type,
                          @Context HttpServletRequest requestContext,
                          @Context HttpServletResponse response,
                          @Suspended AsyncResponse asyncResponse) {
    scheduler.schedule(new Runnable() {
      @Override
      public void run() {
        asyncResponse.resume(...)
      }
    }, 5, TimeUnit.SECOND);
  }
}

This way, no thread is blocked for the waiting time of five seconds what gives an opportunity for handling other requests in the meantime.

JAX-RS does not offer a way of completely discarding a request without an answer. You will need to keep the connection open to produce a time out, if you terminate the connection, a user is notified of the termination. Best you could do would be to never answer an asynchronous request but this will still consume some ressources. If you wanted to avoid this, you would have to solve the problem outside of JAX-RS, for example by proxying the request by another server.

One way to do that would be to set up mod_proxy where you could answer the proxy with an error code for the mallicious requests and set up a very large retry limit for such requests.

Sacrilegious

This is definitely better that Thread.sleep(5000) but resources will still be consumed. As I understood the question, OP is asking for ignoring the request for always, not returning to processing it 5 seconds later. – Agrippina

In this case, it is possible to never resume an async respone but it would be necessary to keep the connection open. Otherwise, the client is notified about the cancelation. However, keeping the connection open still consumes some ressources. – Sacrilegious

Exactly: never resuming the async reponse will consume resources of the Operating System, as the connection remains open. Thus, not a good idea. OP wants to silently drop the connection and all related resources, letting the client connection to time out. Even if you never resume the async response, you will not obtain the desired behaviour. – Agrippina

This is a very good solution. The async response will consume very minimal resources (assuming NIO is being used in the container). Leaving a connection open does not mean assigning a thread to it. I am sure that an average server can sustain at least 10,000 of those requests at a time, which is the best you can get with this setup. Just make sure the scheduler you use is a single thread pool. – Brittnee

If you are using a Grizzly connector, easily several hundre thousand. The question is, how many requests the server needs to handle. – Sacrilegious

Answer 2

I may suggest move IP deny logic from REST to plain HTTP Filter:

@WebFilter(urlPatterns = "/*", asyncSupported = true)
@WebListener
public class HttpFilter implements Filter {

    @Override
   public void init(FilterConfig filterConfig) throws ServletException {   }

    @Override
    public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
        if(denyIP(servletRequest.getRemoteAddr())) {
            AsyncContext asyncContext = servletRequest.startAsync();
            asyncContext.setTimeout(100000);
        }else{
           filterChain.doFilter(servletRequest, servletResponse);
        }
    }

    @Override
    public void destroy() {   }

    private boolean denyIP(String address){
         //todo
         return true;
    }

}

It is cheaper for application server: no XML/JSON deserialization, no call to REST classes. Also you may notice that I never call asyncContext.start. I check Wildfly 8.2 server. In this case Wildfly does not use thread per request. I sent a lot of requests, but amount of threads was constant.

PS

trying to guess application IDs by looping over random IDs

It is not DOS attack. It is brute force attack.

Baucom

"In this case Wildfly does not use thread per request. I sent a lot of requests, but amount of threads was constant." Is that because you've never called asyncContext.start()? – Beltz

Yes. Wildfly adds expiration tasks to queue, and there is thread pool for processing queue, but this thread pool allocates limited amount of threads. – Baucom

Answer 3

There are many possible solutions, with the restrictions you have given 2 possible solutions come to my mind:

1) Use a forward proxy that already has support for limiting requests. I personally have used Nginx and can recommend it partly because it is simple to set up. Relevant rate limiting config: Limit Req Module

2) Use Asynchronous JAX-RS to let timeout the malicious request that you detected. Sane request can be processed directly. But beware of the consequences, either way such an approach will consume resources on the server!

Supinator

Answer 4

You can try asyncContext supported from Tomcat 3.0. This feature decouples web request handler and processer. In your case, the thread which accepts the request has to wait/sleep more than timeout configured. Making the same thread sleep for such a long time would starve them and it would drastically affect the performance of servers. So, asynchronous processing is the right way to go.

I have used asyncContext with Java single thread executor http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html It worked well for me. I had similar business case, where I had to mock my application.

Refer this for implementation http://peter-braun.org/2013/04/asynchronous-processing-from-servlet-3-0-to-jax-rs-2-0/

Single thread executor would not eat resources and its ideal for this use case.

Patriotism

Answer 5

I have not tried this.... just a shot in the dark here, so take it with a grain of salt. So the problem is that once you detect something fishy and put the IP in block-mode you do not want to waste another iota of resources on this request and also cause them to waste time timingout. But your framework will respond if you throw an exception. How about interrupting your current thread? You can do this by Thread.currentThread().interrupt();. The hope is that the Java container processing the request will check the interrupt status. It might not be doing that. I know I have seen IO related classes not process requests because the interrupted flag was set.

EDIT: If interrupting your thread does not work, you can also try throwing an InterruptedException. It might accomplish the desired affect.

Bakker

Answer 6

As far as I know, in the Servlet specification there is no such mechanism. As JAX-RS implementations use servlets (at least Jersey and Resteasy), you do not have a standard way to achieve that in Java.

The idea of using the aynchronous JAX-RS is better than Thread.sleep(5000), but it will still use some resources and is nothing but a way to process the request later, instead of ignoring the request for always.

Agrippina

Here another credible answer: #22669325 – Agrippina

Answer 7

I once solved a similar problem by creating a TCP/IP tunnel application.

It's a small application that listens to an external port (e.g. the http port 80). Under normal conditions, all received calls are accepted, creating dedicated socket objects. These individual sockets then call the real (hidden) webserver, which runs on the same physical server, or any server within your local network.

The tunnel itself is like a dispatcher, but can also act like a loadbalancer. It can be multifunctional.

The thing is that you're working low-level with sockets at this point. If you hand a list of banned ip-addresses to this application, then it can shut-out applications on a very low-level (not even accepting their socket calls at all).

You could integrate this in the very same java application as well. But I think it is more flexible to regard this as a seperate application.

(Let me know if you need some source code, I may have some code laying around to get you started)

Impious

I realize that this answer is a bit far-fetched, and I guess some people may want to down-vote it. But since nobody else mentioned this approach, ... I just didn't want you to overlook this option. It's just an open-minded out-of-the-box idea. :) I hope you don't hate it. – Impious