Java NIO: transferFrom until end of stream

Question

I'm playing around with the NIO library. I'm attempting to listen for a connection on port 8888 and once a connection is accepted, dump everything from that channel to somefile.

I know how to do it with ByteBuffers, but I'd like to get it working with the allegedly super efficient FileChannel.transferFrom.

This is what I got:

ServerSocketChannel ssChannel = ServerSocketChannel.open();
ssChannel.socket().bind(new InetSocketAddress(8888));

SocketChannel sChannel = ssChannel.accept();
FileChannel out = new FileOutputStream("somefile").getChannel();

while (... sChannel has not reached the end of the stream ...)     <-- what to put here?
    out.transferFrom(sChannel, out.position(), BUF_SIZE);

out.close();

So, my question is: How do I express "transferFrom some channel until end-of-stream is reached"?

---

^{Edit: Changed 1024 to BUF_SIZE, since the size of the buffer used, is irrelevant for the question.}

Marty

Answer 1

There are few ways to handle the case. Some background info how trasnferTo/From is implemented internally and when it can be superior.

• 1st and foremost you should know how many bytes you have to xfer, i.e. use FileChannel.size() to determine the max available and sum the result. The case refers to FileChannel.trasnferTo(socketChanel)
• The method does not return -1
• The method is emulated on Windows. Windows doesn't have an API function to xfer from filedescriptor to socket, it does have one (two) to xfer from the file designated by name - but that's incompatible with java API.
• On Linux the standard sendfile (or sendfile64) is used, on Solaris it's called sendfilev64.

in short for (long xferBytes=0; startPos + xferBytes<fchannel.size();) doXfer() will work for transfer from file -> socket. There is no OS function that transfers from socket to file (which the OP is interested in). Since the socket data is not int he OS cache it can't be done so effectively, it's emulated. The best way to implement the copy is via standard loop using a polled direct ByteBuffer sized with the socket read buffer. Since I use only non-blocking IO that involves a selector as well.

That being said: I'd like to get it working with the allegedly super efficient "? - it is not efficient and it's emulated on all OSes, hence it will end up the transfer when the socket is closed gracefully or not. The function will not even throw the inherited IOException, provided there was ANY transfer (If the socket was readable and open).

I hope the answer is clear: the only interesting use of File.transferFrom happens when the source is a file. The most efficient (and interesting case) is file->socket and file->file is implemented via filechanel.map/unmap(!!).

Sundew

"Once a connection is accepted, dump everything from that channel to somefile." So the input is a socket channel, so he can't possibly know how much data is in it, unless the sender sends a length word first. I don't know what "use FileChannel.size() to determine the max available and sum the result" means. – Icebound

@EJP - read further, it does tell socket -> file is futile and emulated. Using a dedicated direct ByteBuffer sized to the socket read buffer is the way to carry the task. 1024 is a very bad size for reading (the allocated memory is always > pageSize [4k], so 1024 just wastes memory and involves more kernel trips). The bullets ahead show how transfer method internally. – Sundew

Of course but the first part of your answer about file->socket remains irrelevant to the OP's question. – Icebound

@EJP, file->socket is the only really useful transfer, so I put it 1st. file->file is almost ok but involves mmap/munmap and the latter is an expensive operation flushing the TLBs of all CPUs (which do suck for multi-CPU and, so calling it has to be done w/ the largest possible length, i.e. it can have even negative overall performance effects). Well ok, I will edit to show the bullets provide background on xfer methods. :) – Sundew

This is all off topic. The user has specified his requirements. – Icebound

Answer 2

I'm not sure, but the JavaDoc says:

An attempt is made to read up to count bytes from the source channel and write them to this channel's file starting at the given position. An invocation of this method may or may not transfer all of the requested bytes; whether or not it does so depends upon the natures and states of the channels. Fewer than the requested number of bytes will be transferred if the source channel has fewer than count bytes remaining, or if the source channel is non-blocking and has fewer than count bytes immediately available in its input buffer.

I think you may say that telling it to copy infinite bytes (of course not in a loop) will do the job:

out.transferFrom(sChannel, out.position(), Integer.MAX_VALUE);

So, I guess when the socket connection is closed, the state will get changed, which will stop the transferFrom method.

But as I already said: I'm not sure.

Reikoreilly

You beat me to it. However you can use Long.MAX_VALUE instead of Integer.MAX_VALUE. – Flabby

Right, but what if I only get like 3 bytes out of 10 in the first call to transferFrom? – Marty

Maybe test your source channel: it has an int read(ByteBuffer) method. If that returns -1 then there's nothing in it, thus you transferred everything into the FileChannel. If you want something more robust, then you are pretty much better of with ByteBuffers (re-using those are pretty much efficient anyway). – Flabby

So what if it doesn't return -1? Then I've consumed a few bytes... That messes up the code completely. – Marty

ReadableByteChannel's documentation states that the method int read(ByteBuffer) returns The number of bytes read, possibly zero, or -1 if the channel has reached end-of-stream. So, if it does return -1, then you've reached the end of the stream. Which should indicate... the end of the stream? If you can't rely on this, then how do you intend to transfer everything from the channel to somewhere else? Other than this, you've wrote that you know how to do it with ByteBuffers. If you know that, then in that case how do you decide if you can stop reading from the channel? – Flabby

Yeah ok, I didn't read your last comment carefully enough: you're right. But the last part of my last comment still stands. :) – Flabby

Sure. I agree that falling back on read would solve the problem of finding out if the end of the stream is reached... Unless it's proved to be the only way, this is not what I'm after though... – Marty

Answer 3

Answering your question directly:

while( (count = socketChannel.read(this.readBuffer) )  >= 0) {
   /// do something
}

But if this is what you do you do not use any benefits of non-blocking IO because you actually use it exactly as blocking IO. The point of non-blocking IO is that 1 network thread can serve several clients simultaneously: if there is nothing to read from one channel (i.e. count == 0) you can switch to other channel (that belongs to other client connection).

So, the loop should actually iterate different channels instead of reading from one channel until it is over.

Take a look on this tutorial: http://rox-xmlrpc.sourceforge.net/niotut/ I believe it will help you to understand the issue.

Biblio

But the call can return 0 bytes without having reached end of stream, right? – Marty

@Marty Only (a) in non-blocking mode, when you should be using a Selector anyway, not a loop, or (b) if the buffer length is zero, which is a programming error. – Icebound

Answer 4

Building on top of what other people here have written, here's a simple helper method which accomplishes the goal:

public static void transferFully(FileChannel fileChannel, ReadableByteChannel sourceChannel, long totalSize) {
    for (long bytesWritten = 0; bytesWritten < totalSize;) {
        bytesWritten += fileChannel.transferFrom(sourceChannel, bytesWritten, totalSize - bytesWritten);
    }
}

Goody

Answer 5

transferFrom() returns a count. Just keep calling it, advancing the position/offset, until it returns zero. But start with a much larger count than 1024, more like a megabyte or two, otherwise you're not getting much benefit from this method.

EDIT To address all the commentary below, the documentation says that "Fewer than the requested number of bytes will be transferred if the source channel has fewer than count bytes remaining, or if the source channel is non-blocking and has fewer than count bytes immediately available in its input buffer." So provided you are in blocking mode it won't return zero until there is nothing left in the source. So looping until it returns zero is valid.

EDIT 2

The transfer methods are certainly mis-designed. They should have been designed to return -1 at end of stream, like all the read() methods.

Icebound

The method's documentation says that it returns The number of bytes, possibly zero, that were actually transferred. This doesn't mean that it can't return with 0 immediately after the first call without transferring anything. (Another question is: when and why would it do that?) However the OP can't rely on this. I'm starting to think that the only way to know for sure that this method transfers everything from a channel to a file is to know in advance how many bytes will be transferred. – Flabby

This was already discussed in the comments section for this answer. However the OP wants an almost one-liner solution (I can relate to that, because it would be awesome). – Flabby

@EJP, do you have any argument or good reference for the fact that falling back on an ordinary read (and possibly having to write such byte "manually") is indeed the best solution to this problem? – Marty

@Marty Nobody said it was the best solution. On investigation of the source, for a SocketChannel the JVM does the transfer itself, so the technique is pointless for SocketChannels. The JVM implements it as between two FileChannels too much to my surprise (ref JDK 1.6 src/share/classes/sun/nio/ch/FileChannelImpl.java), so the Javadoc statement about the OS 'transfer[ring] bytes directly ... without actually copying them' is 'inoperative'. – Icebound

@KohányiRóbert You can do this in two lines of code if you don't mind omitting the braces. You can't use transferTo() or transferFrom() in one line, whatever the OP may want. – Icebound

Perhaps the downvote is due to the fact that it would be wrong to call transferFrom until it returns 0 if the intention is to deplete the stream (as pointed out by @KohányiRóbert). Unless one reads the comments, the "Just keep calling it, advancing the position/offset, until it returns zero." is actually quite misleading. – Marty

@Marty That wouldn't be wrong at all, see my edit. Most of the commentary above is misdirected. – Icebound

@KohányiRóbert To address your first comment above "this doesn't mean that it can't return with 0", in blocking mode it does mean exactly that. – Icebound

I made the same yesterday and believe this is the correct way of doing it. Also, implementing it this way enables reporting the status of the transfer and that might come in handy. – Uranography

Answer 6

allegedly super efficient FileChannel.transferFrom.

If you want both the benefits of DMA access and nonblocking IO the best way is to memory-map the file and then just read from the socket into the memory mapped buffers.

But that requires that you preallocate the file.

Manure

Answer 7

This way:

URLConnection connection = new URL("target").openConnection();
File file = new File(connection.getURL().getPath().substring(1));
FileChannel download = new FileOutputStream(file).getChannel();

while(download.transferFrom(Channels.newChannel(connection.getInputStream()),
        file.length(), 1024) > 0) {
    //Some calculs to get current speed ;)
}

Alisiaalison