java.net.SocketException: Too many open files

Question

I have a java app which runs just fine (on Ubuntu 10.04) for few hours until it hits "java.net.SocketException: Too many open files". The code for Sender.java can be found here

Is it because I create a new instance of HttpPut and HttpPost for each thread? I'm using apache-commons HTTPClient 4.

Here's the exception log:

java.net.SocketException: Too many open files
    at java.net.Socket.createImpl(Socket.java:414)
    at java.net.Socket.connect(Socket.java:544)
    at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:123)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:133)
    at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
    at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:108)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
    at com.marketplace.io.Sender.doBasicHttpPost(Sender.java:434)
    at com.marketplace.io.Sender.appVisualExists(Sender.java:223)
    at com.marketplace.io.Sender.addVisualToCollection(Sender.java:350)
    at com.marketplace.service.ImageThread.run(ImageThread.java:136)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:636)

Hunkydory

Do you ever shutdown your httpclient? – Offstage

I don't have time to look through all that code (sorry) but it sounds like you're (continually) not closing sockets that you open. – Erective

@The Elite Gentleman- No I don't. I'm not sure how to do that. Some of the documentation for HTTPClient is non-existant. – Hunkydory

I'm going to try httpClient.getConnectionManager().shutdown(); and see if that has any effect. – Hunkydory

bingo! That's how you close your httpclient's connection manager. – Offstage

Answer 1

On line 438 you get the response as a stream and convert that to a byte array. The InputStream returned by entity.getContent() does not get closed. This could be contributing to the problem. Also, the HttpEntity.consumeContent() is deprecated for related reasons.

Cassino

Thanks for going into the trouble and looking at the source code. I'm going to leave the code running overnight and see if your suggested changes make any difference. – Hunkydory

I'm not sure what fixed the problem; whether it was httpClient.getConnectionManager().shutdown(); or your suggestion. Thanks for your help. – Hunkydory

Answer 2

"java.net.SocketException: Too many files open"can be seen any Java Server application e.g. Tomcat, Weblogic, WebSphere etc, with client connecting and disconnecting frequently.

Please note that socket connections are treated like files and they use file descriptor, which is a limited resource.

Different operating system has different limits on number of file handles they can manage.

In short, this error is coming because clients are connecting and disconnecting frequently.If you want to handle it on your side, you have two options :

1) Increase number of open file handles or file descriptors per process.

In UNIX based operating system e.g. Ubuntu or Solaris, you can use command ulimit -a to find out how many open file handles per process is allowed.

$ ulimit -a
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 256
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 2048
virtual memory        (kbytes, -v) unlimited

You can see that, open files (-n) 256, which means only 256 open file handles per process is allowed. If your Java program, remember Tomcat, weblogic or any other application server are Java programs and they run on JVM, exceeds this limit, it will throw java.net.SocketException: Too many files open error.

You can change this limit by using ulimit -n to a larger number e.g. 4096, but do it with advise of UNIX system administrator and if you have separate UNIX support team, than better escalate to them.

2) Reduce timeout for TIME_WAIT state in your operating system

In UNIX based systems, you can see current configuration in /proc/sys/net/ipv4/tcp_fin_timeout file.

In Windows based system, you can see this information in windows registry. You can change the TCPTIME_WAIT timeout in Windows by following below steps :

1) Open Windows Registry Editor, by typing regedit in run command window
2) Find the key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\tcpip\Parameters
3) Add a new key value pair TcpTimedWaitDelay asa decimal and set the desired timeout in seconds (60-240)
4) Restart your windows machine.

Dittography

I used root to login and executed these commands. But when I switched to other ordinary user, I found the open files number hadn't been changed. Could I configure the number of open file handles for all users? – Millennial

Verify your setting by: tail -f /etc/security/limits.conf. Please make sure your changes has been saved so that it can be applied for all users – Dittography

With verifying the setting by: ulimit -n, it seems it's been changed from 1024 to 4096(after running ulimit -n 4096). But bying tail -f /etc/security/limits.conf, I found there isn't any changes in the file. – Millennial

Please edit limits.conf file using: vi /etc/security/limits.conf Append the two rows below to the file to set soft and hard limits for all users (* means apply for all users): * soft nofile 4096 * hard nofile 10240 – Dittography

If I appended these two lines by editing limits.conf file, the result of running ulimit -n was still the preceding value 1024. Is it correct? Should I restart my service after editing this file? – Millennial

Answer 3

You might also want to check the linux maximum open file limit. This related link is for a bespoke java-based product but it nicely explains the steps required to fix the issue.

Patrilocal

And, a definition for the most annoying tech term since 'cloud': techterms.com/definition/bespoke – Ionization

Can I get reputation points for correct usage of annoying terms? ;) – Patrilocal

Answer 4

(RESOLVED)

I recently have the same error because the var/log on DataBase Server is Full.

#df -h
S.ficheros            Size  Used Avail Use% Montado en

/dev/cciss/c0d0p3     126G  126G    0G 100% /var


#echo "">/var/log/postgresql/postgresql.log

Now, the error is gone!!!

important: see what to log in postgresql.log, review your postgresql.conf

bye
@_jpgo

Centesimal

Answer 5

I solve the problem by closing the connection in the finally block。

public  static String postMethod(String jsonStr,String sendUrl){
    String resultJson = "";
    HttpClient httpClient = new HttpClient();
    PostMethod post = new PostMethod(sendUrl);
    post.setRequestHeader("Content-Type","application/x-www-form-urlencoded;charset=utf-8");  
    NameValuePair[] param = { new NameValuePair("message",jsonStr)} ;
    post.setRequestBody(param);
    try {
        httpClient.executeMethod(post);
        resultJson = post.getResponseBodyAsString();
    } catch (HttpException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }finally{
         post.releaseConnection();
         ((SimpleHttpConnectionManager)httpClient.getHttpConnectionManager()).shutdown();
    }
    return resultJson;
}

Disillusion