How can AsynchronousFileChannel read large file?

Question

  Path file = Paths.get("c:/large.log");
  AsynchronousFileChannel channel = AsynchronousFileChannel.open(file);
  final ByteBuffer buffer = ByteBuffer.allocate(1000);
  channel.read(buffer, 0, buffer,
      new CompletionHandler<Integer, ByteBuffer>() {
        public void completed(Integer result, ByteBuffer attachment) {
          System.out.println(new String(buffer.array()));
        }
  }); 

In this way, I can read the first 1000 byte from large.log. How can I read following log If I don't want to allocate bigger byte array like ByteBuffer.allocate(1000*1000). Because I think this will lead to OutOfMemory.

Could someone give me the sample code? Thanks.

ps:I can loop read the large file with JIO because I can check the return value of java.io.BufferedReader.read(). But I don't know how to do with NIO2.

Pinky

Answer 1

Here's a hack that works.

A couple of things that you'll want to note:

1. I've just used your buffer.array() for the output. I had to use buffer.clear() to reset the position so that the asynchronous read will see that there are 1000 spare bytes, but this doesn't clear the existing data out of the array. As a result when you're at the end of the file, if you read fewer than 1000 bytes it prints the whole buffer: however much you just read, plus the remaining 1000 bytes of whatever was last in the end of the buffer. In real life you'd want to do something about that (perhaps with result or with the position of the buffer.
2. For reasons I couldn't figure out buffer which is a class variable is fine within the completed method, but channel which is also a class variable is null. I haven't yet figured out why that would be. So I changed it so it passes channel as the attachment rather than buffer. Still makes no sense to me.
3. The asynchronous read thread isn't important enough to keep the jvm running. So I've simply put a read at the end of the main method. Press Enter to exit.
4. Class variable pos maintains the position in the file you're reading from.
5. The magic happens when you initiate another asynchronous read during the complete method. This is why I discarded the anonymous class and implemented the interface itself.
6. You'll want to switch the path back to yours.

Have fun.

import java.nio.*;
import java.nio.channels.*;
import java.nio.file.*;
import java.io.IOException;

public class TryNio implements CompletionHandler<Integer, AsynchronousFileChannel> {

       // need to keep track of the next position.
        int pos = 0;
        AsynchronousFileChannel channel =  null;
        ByteBuffer buffer = null;

        public void completed(Integer result, AsynchronousFileChannel attachment) {
                 // if result is -1 means nothing was read.
                if (result != -1) {
                        pos += result;  // don't read the same text again.
                                        // your output command.
                        System.out.println(new String(buffer.array()));

                        buffer.clear();  // reset the buffer so you can read more.
                }
                        // initiate another asynchronous read, with this.
                attachment.read(buffer, pos , attachment, this );


        }
        public void failed(Throwable exc,
                        AsynchronousFileChannel attachment) {
                System.err.println ("Error!");
                exc.printStackTrace();
        }

        public void doit() {
                Path file = Paths.get("/var/log/syslog");
                AsynchronousFileChannel channel =  null;
                try {
                        channel = AsynchronousFileChannel.open(file);
                } catch (IOException e) {
                        System.err.println ("Could not open file: " + file.toString());
                        System.exit(1); // yeah.  heh.
                }
                buffer = ByteBuffer.allocate(1000);

                 // start off the asynch read. 
                channel.read(buffer, pos , channel, this );
                // this method now exits, thread returns to main and waits for user input.
        }

        public static void main (String [] args) {
                TryNio tn = new TryNio();
                tn.doit();
             // wait fur user to press a key otherwise java exits because the 
             // asynch thread isn't important enough to keep it running.
                try { System.in.read(); } catch (IOException e) { }
        }
}

Carbazole

I have executed this sample code. But it print more content than what the actual log file have. – Pinky

System.out.print(new String(buffer.array(),0,result));Other the junk data will be printed. – Pinky

Exactly, and note that this problem was present in the OP's orginal code. – Ardrey

Want to point out that the above approach to reading a file looks very inefficient to me. It's recursively spawning a new thread with each asynchronous call. If you want to read a large file asynchronously (not a bad idea), it would make much more sense to spawn just a single thread to do the job, i.e., with plain old Runnable and join(), etc. Maybe there's a newer, better way to do it, but the above code doesn't look like a good way to me. – Dispread

Thinking a little more, I can imagine situations where you'd want to spawn multiple, simultaneous threads to read a single file. But, the above code synchronously spawns the async read threads, waiting for each to complete before spawning a new one. Again, doesn't seem to make sense. Something like this makes more sense to me. – Dispread

The reason why channel must have been null is because the completion handler gets executed in a different thread & as per java memory model, there is no guarantee for other threads to see the latest state unless you explicitly mark it as volatile/protect it by locks/VarHandles. Just to test, mark your class level fields as volatile & see if they are still null (they should not be). Note that volatile is too coarse grained tool. If performance is the criteria, you would want to use VarHandles – Adverse

Answer 2

The GregHNZ solution is great and since I have to use this kind of code several times in different projects, I ended up putting it in an auxiliary library RxIo which I published in Maven Central Repository and is also available at RxIo github repository. With RxIo you can use the RxIo utility class to read all bytes of a file like:

AsyncFiles
    .readAllBytes(Paths.get("input.txt"))
    .thenApply(bytes -> { /*... use bytes... */});

The readAllBytes(Path file) allocates a ByteBuffer with a default size of 262144, but you can specify a different value using the readAllBytes(Path file, int bufferSize).

You can see other use cases in unit tests folder.

Docket

Answer 3

Start another read in the completionHandler if there is anything left in the file. But I would use a much bigger buffer than 1000, at least 8192.

Ardrey

Works for me. Of course you have to clear the buffer, and increment the position parameter in the read, so there is a bit of mucking around with final variables, but it can be done. – Ardrey

Answer 4

Using position in the file and file size, asynchronous read operation needs to be repeatedly called in the completionhandler to read entire file. Position needs to be increased using the number of bytes red as read operation completes each time.

Below is completed method of completionhandler to asynchronously read entire file. For complete example see http://www.zoftino.com/java-asynchronous-io-nio2

public void completed(Integer result, ByteBuffer attachment) {
    try {
        bb.flip();
        System.out.println("bytea red "+bb.limit());

        if(afc.size() > position) {
            position = position + bb.limit();
            bb.clear();
            //pass the same completion handler
            afc.read(bb, position, bb, this);
        }
    } catch (IOException e) {
        e.printStackTrace();
    }       
}

Polemics