reduce number of opened files in java code

Question

Hi I have some code that uses block

RandomAccessFile file = new RandomAccessFile("some file", "rw");
FileChannel channel = file.getChannel();

// some code
String line = "some data";
ByteBuffer buf = ByteBuffer.wrap(line.getBytes());
channel.write(buf);

channel.close();
file.close();

but the specific of the application is that I have to generate large number of temporary files, more then 4000 in average (used for Hive inserts to the partitioned table).

The problem is that sometimes I catch exception

Failed with exception Too many open files

during the app running.

I wounder if there any way to tell OS that file is closed already and not used anymore, why the

channel.close();
file.close();

does not reduce the number of opened files. Is there any way to do this in Java code?

I have already increased max number of opened files in

#/etc/sysctl.conf:
kern.maxfiles=204800
kern.maxfilesperproc=200000
kern.ipc.somaxconn=8096

Update: I tried to eliminate the problem, so I parted the code to investigate each part of it (create files, upload to hive, delete files).

Using class 'File' or 'RandomAccessFile' fails with the exception "Too many open files".

Finally I used the code:

FileOutputStream s = null;
FileChannel c = null;

try {
    s = new FileOutputStream(filePath);
    c = s.getChannel();
    // do writes
    c.write("some data"); 
    c.force(true);
    s.getFD().sync();

} catch (IOException e) {
    // handle exception
} finally {
    if (c != null)
        c.close();
    if (s != null)
        s.close();
}

And this works with large amounts of files (tested on 20K with 5KB size each). The code itself does not throw exception as previous two classes. But production code (with hive) still had the exception. And it appears that the hive connection through the JDBC is the reason of it. I will investigate further.

Depict

Sounds like something is relying on the finalizer to close external resources. That's… asking for trouble really. – Solfa

Answer 1

The amount of open file handles that can be used by the OS is not the same thing as the number of file handles that can be opened by a process. Most unix systems restrict the number of file handles per process. Most likely it something like 1024 file handles for your JVM.

a) You need to set the ulimit in the shell that launches the JVM to some higher value. (Something like 'ulimit -n 4000')

b) You should verify that you don't have any resource leaks that are preventing your files from being 'finalized'.

Gravitative

Answer 2

Make sure to use a finally{} block. If there is an exception for some reason the close will never happen in the code as written.

Concertino

Answer 3

Is this the exact code? Because I can think of one scenario where you might be opening all the files in a loop and written the code to close all of them in the end which is causing this problem. Please post the full code.

Calumniate

usage patter is the following: 1) get data from mysql source to some data objects 2) loop the list of objects and in each iteration do the posted code. So I do the posted peace of code in each iteration and do not have all of the files opened once. – Depict