How to stop hack/DOS attack on web API

Question

My website has been experiencing a denial of service/hack attack for the last week. The attack is hitting our web API with randomly generated invalid API keys in a loop.

I'm not sure if they are trying to guess a key (mathematically impossible as 64bit keys) or trying to DOS attack the server. The attack is distributed, so I cannot ban all of the IP address, as it occurs from hundreds of clients.

My guess is that it is an Android app by the IPs, so someone has some malware in an Android app, and use all the installs to attack my server.

Server is Tomcat/Java, currently the web API just responds 400 to invalid keys, and caches IPs that have made several invalid key attempts, but still needs to do some processing for each bad request.

Any suggestions how to stop the attack? Is there any way to identify the Android app making the request from the HTTP header?

Somatology

What's the HTTP header? – Stereochemistry

the header has the IP, the agent string is null – Somatology

mathematically impossible as 64bit keys Wut? a) No encryption system with a key shorter than the message is able prevent an attacker from getting some hidden information (worst case, the whole plaintext). There is no "Impossible" here. b) 64bit keys (DES?) in 2015 is not much. I'd rather say it's weak. – Ahearn

The hacker is trying to guess an app ID by randomly guessing IDs, our app IDs are 64 bit longs, so your chance of guessing a valid app ID is about 1 in 18,446,744,073,709,551,616. So if you made 1,000 attempts per second, it would take you about 584,942,417 years to guess a valid ID – Somatology

Well, but trying HTTP requests and trying to decrypt locally available data is different. With latter, and a portion cloud/GPU stuff, brute-forcing it gets realistic (even single units which can crack eg. DES in less than a day exist) – Ahearn

I'm pretty certain the hack is coming from a malware Android app, seems like Android is the ideal platform for hackers now, no security restrictions like a browser, mass deployment in a game or chat app makes attacking any server easy. There should be some way to determine the app a request comes from, otherwise Google is just letting hacker take over. – Somatology

Take a look at OWASP: owasp.org/index.php/Web_Service_Security_Cheat_Sheet – Peak

I added the delay in responding to a bad request, and that seems to have at least reduced the number of requests, looking into just closing the request next – Somatology

Does anyone get my point that JavaScript added support to limit dos attacks by disabling cross domain access, but Android has no such security, so make dos and hacking easy. – Somatology

I don't know how big your user base is for the web API but setting up a whitelist should fix your problem if it is a manageable amount of users. That is all advice I can give, unfortunately. – Mumps

How can you tell it's coming from an Android app just by knowing the IP? – Agrarian

Answer 1

Preventing Brute-Force Attacks:

There is a vast array of tools and strategies available to help you do this, and which to use depends entirely on your server implementation and requirements.

Without using a firewall, IDS, or other network-control tools, you can't really stop a DDOS from, well, denying service to your application. You can, however, modify your application to make a brute-force attack significantly more difficult.

The standard way to do this is by implementing a lockout or a progressive delay. A lockout prevents an IP from making a login request for X minutes if they fail to log in N times. A progressive delay adds a longer and longer delay to processing each bad login request.

If you're using Tomcat's authentication system (i.e. you have a <login-constraint> element in your webapp configuration), you should use the Tomcat LockoutRealm, which lets you easily put IP addresses on a lockout once they make a number of bad requests.

If you are not using Tomcat's authentication system, then you would have to post more information about what you are using to get more specific information.

Finally, you could simply increase the length of your API keys. 64 bits seems like an insurmountably huge keyspace to search, but its underweight by modern standards. A number of factors could contribute to making it far less secure than you expect:

• A botnet (or other large network) could make tens of thousands of attempts per second, if you have no protections in place.
• Depending on how you're generating your keys and gathering entropy, your de facto keyspace might be much smaller.
• As your number of valid keys increases, the number of keys that need to be attempted to find a valid one (at least in theory) drops sharply.

Upping the API key length to 128 (or 256, or 512) won't cost much, and you'll tremendously increase the search space (and thus, the difficulty) of any brute force attack.

Mitigating DDOS attacks:

To mitigate DDOS attacks, however, you need to do a bit more legwork. DDOS attacks are hard to defend against, and its especially hard if you don't control the network your server is on.

That being said, there are a few server-side things you can do:

• Installing and configuring a web-application firewall, like mod_security, to reject incoming connections that violate rules that you define.
• Setting up an IDS system, like Snort, to detect when a DDOS attack is occurring and take the first steps to mitigate it
• See @Martin Muller's post for another excellent option, fail2ban
• Creating your own Tomcat Valve, as described here, to reject incoming requests by their User-Agents (or any other criterion) as a last line of defense.

In the end, however, there is only so much you can do to stop a DDOS attack for free. A server has only so much memory, so many CPU cycles, and so much network bandwidth; with enough incoming connections, even the most efficient firewall won't keep you from going down. You'll be better able to weather DDOS attacks if you invest in a higher-bandwidth internet connection and more servers, or if you deploy your application on Amazon Web Services, or if you bought one of many consumer and enterprise DDOS mitigation products (@SDude has some excellent recommendations in his post). None of those options are cheap, quick, or easy, but they're what's available.

Bottom Line:

If you rely on your application code to mitigate a DDOS, you've already lost

Puryear

Thanks, I thought about delaying the response to a bad IP, not sure if more server time would be spent sleeping the response thread than just rejecting the request. Maybe just rejecting the request uses the least server resources, but the requests from the hacker are at 20,000 requests and growing. Is there a way to just not answer a request, so the client hangs, at least for a while? – Somatology

Absolutely. HTTP has no way of knowing if there's anyone on the other end until it gets a response. If you don't respond, the client will just wait until its timeout elapses. Overall, however, DDOS attacks are hard to defend against, especially if you can't control the network. I would highly recommend using a lower-level solution, like a properly configured firewall, rather than try to bake DDOS mitigation techniques into your app. If they're connecting to your app and using even small amounts of resources, they're winning. – Puryear

@James, one lower-level method would be to write a Tomcat Valve to reject requests with a null user-agent; this thread has some good information on how to do so. – Puryear

Thanks, I can't see how the firewall can help as the attacks come from hundreds/unknown IP and are executing validate web API requests (just with invalid API keys). Any idea how to not respond to a request in a Tomcat REST service? – Somatology

We support a web API, and support Android and JavaScript clients, so must allow null agents. – Somatology

@James, what library are you using for handling HTTP connections? In many cases, you can simply call the equivalent close() method on the object representing an incoming connection, and the object will die without sending anything down the wire. I must stress, however, that this is not a good way to deal with a DDOS attack; you need a network level solution. A firewall would be best used in combination with something like Snort (mentioned above) to create a dynamically generated IP blacklist (or something of that ilk), – Puryear

@James, If you really, really can't set up a firewall, this post has some tips for in-application techniques that would help somewhat. – Puryear

Thanks for all the details, the keys are actually free for a user to create (and user accounts are also free), so 64bit is enough (which is why it is odd the hacker is going to such efforts to break a free API key...) – Somatology

I added the delay in responding to a bad request, and that seems to have at least reduced the number of requests, looking into just closing the request next – Somatology

I'm trying to implement just closing the connection, or not sending a response, to leave the client waiting so it can't send its next request, anyone have any idea how to not send a response for a JAX RS service (i.e. @Path) – Somatology

Answer 2

If it's big enough you just can't stop it alone. You can do all the optimisation you want at the app level, but you'll still go down. In addition to app-level security for prevention (as in FSQ's answer) you should use proven solutions leaving the heavy lifting to professionals (if you are serious about your business). My advise is:

1. Sign-up for CloudFlare or Incapsula. This is day to day for them.
2. Consider using AWS API gateway as the second stage for your API requests. You'll enjoy filtering, throttling, security,auto-scaling and HA for your API at Amazon scale. Then you can forward the valid requests to your machines (in or outside amazon)

Internet --> CloudFlare/Incapsula --> AWS API Gateway --> Your API Server

0,02

PS: I think this question belongs to Sec

Shig

Why do we need both CloudFlare and AWS API gateway? – Nonpros

Answer 3

Here are a couple ideas. There are a number of strategies in addition, but this should get you started. Also realize that amazon gets ddos'd on a frequent basis and their systems tend to have a few heuristics that harden them (and therefore you) from these attacks, particularly if you are using Elastic load balancing, which you should be using anyway.

• Use a CDN -- they often have ways of detecting and defending against ddos. Akamai, mastery, or amazons own cloud front.
• Use iptables to blacklist offensive ips. The more tooling you have around this, the faster you can blok/unblock

• Use throttling mechanisms to prevent large numbers of requests

• Automatically deny requests that are very large (say greater than 1-2mb; unless you have a photo uploading service or similar) before they get to your application

• Prevent cascading failures by placing a limit on the total number of connections to other components in your system; for example, dont let your database server become overloaded by opening a thousand connections to it.

Massotherapy

Answer 4

The best way is to prevent the access to your services entirely for those IP addresses who have failed let's say 3 times. This will take most of the load from your server as the attacker gets blocked before Tomcat even has to start a thread for this user.

One of the best tools to achieve this is called fail2ban (http://www.fail2ban.org). It is provided as a package in all major linux distributions.

What you have to do is basically log the failed attempts into a file and create a custom filter for fail2ban. Darryn van Tonder has a nice example on how to write your own filter on his blog: https://darrynvt.wordpress.com/tag/custom-fail2ban-filters/

Arlindaarline

Answer 5

If D-DOS is attack is severe, application level checks does not work at all. Entire bandwidth will be consumed by D-DOS clients and your application level checks won't be triggered. Practically your web service does not run at all.

If you have to keep your application safe from severe D-DOS attacks, you do not have any other option except relying on third party tools by paying money. One of the Clean pipe provider ( who sends only good traffic) tools I can bank on from my past experience : Neustar

If D-DOS attack is mild in your website, you can implement application level checks. For example, below configuration will restrict maximum number of connections from single IP as quoted in Restrict calls from single IP

<Directory /home/*/public_html> -- You can change this location
    MaxConnPerIP 1  
    OnlyIPLimit audio/mpeg video
</Directory>

For more insight into D-DOS attack, visit Wiki link. It provides list of preventive & responsive tools which includes : Firewalls, Switches, Routers, IPs Based Prevention, D-DOS based defences

and finally

Clean pipes (All traffic is passed through a "cleaning center" or a "scrubbing center" via various methods such as proxies, tunnels or even direct circuits, which separates "bad" traffic (DDoS and also other common internet attacks) and only sends good traffic beyond to the server)

You can find 12 distributors of Clean pipes.

Ridgeway

Thanks, it is not sever yet, hopefully it does not come to this, I doubt I could afford it. – Somatology

In that case, you can go for IPs based preventions, Firewalls etc. – Ridgeway

Answer 6

For a targeted and highly distributed DOS attack the only practical solution (other than providing the capacity to soak it up) is to profile the attack, identify the 'tells' and route that traffic to a low resource handler.

Your question has some tells - that the request is invalid, but presumably there is too much cost in determining that. That the requests originate from a specific group of networks and that presumably they occur in bursts.

In your comments you've told us at least one other tell - the user agent is null.

Without adding any additional components, you could start by tarpitting the connection - if a request matching the profile comes in, go ahead and validate the key, but then have your code sleep for a second or two. This will reduce the rate of requests from these clients at a small cost.

Another solution would be to use log failures matching the tell and use fail2ban to reconfigure your firewall in real time to drop all packets from the source address for a while.

No, its unlikely you will be able to identify the app without getting hold of an affected device.

Edmea