InputStream or Reader wrapper for progress reporting

Question

So, I'm feeding file data to an API that takes a Reader, and I'd like a way to report progress.

It seems like it should be straightforward to write a FilterInputStream implementation that wraps the FileInputStream, keeps track of the number of bytes read vs. the total file size, and fires some event (or, calls some update() method) to report fractional progress.

(Alternatively, it could report absolute bytes read, and somebody else could do the math -- maybe more generally useful in the case of other streaming situations.)

I know I've seen this before and I may even have done it before, but I can't find the code and I'm lazy. Has anyone got it laying around? Or can someone suggest a better approach?

---

One year (and a bit) later...

I implemented a solution based on Adamski's answer below, and it worked, but after some months of usage I wouldn't recommend it. When you have a lot of updates, firing/handling unnecessary progress events becomes a huge cost. The basic counting mechanism is fine, but much better to have whoever cares about the progress poll for it, rather than pushing it to them.

(If you know the total size, you can try only firing an event every > 1% change or whatever, but it's not really worth the trouble. And often, you don't.)

Zackzackariah

Answer 1

Here's a fairly basic implementation that fires PropertyChangeEvents when additional bytes are read. Some caveats:

• The class does not support mark or reset operations, although these would be easy to add.
• The class does not check whether the total number bytes read ever exceeds the maximum number of bytes anticipated, although this could always be dealt with by client code when displaying progress.
• I haven't test the code.

Code:

public class ProgressInputStream extends FilterInputStream {
    private final PropertyChangeSupport propertyChangeSupport;
    private final long maxNumBytes;
    private volatile long totalNumBytesRead;

    public ProgressInputStream(InputStream in, long maxNumBytes) {
        super(in);
        this.propertyChangeSupport = new PropertyChangeSupport(this);
        this.maxNumBytes = maxNumBytes;
    }

    public long getMaxNumBytes() {
        return maxNumBytes;
    }

    public long getTotalNumBytesRead() {
        return totalNumBytesRead;
    }

    public void addPropertyChangeListener(PropertyChangeListener l) {
        propertyChangeSupport.addPropertyChangeListener(l);
    }

    public void removePropertyChangeListener(PropertyChangeListener l) {
        propertyChangeSupport.removePropertyChangeListener(l);
    }

    @Override
    public int read() throws IOException {
        int b = super.read();
        updateProgress(1);
        return b;
    }

    @Override
    public int read(byte[] b) throws IOException {
        return (int)updateProgress(super.read(b));
    }

    @Override
    public int read(byte[] b, int off, int len) throws IOException {
        return (int)updateProgress(super.read(b, off, len));
    }

    @Override
    public long skip(long n) throws IOException {
        return updateProgress(super.skip(n));
    }

    @Override
    public void mark(int readlimit) {
        throw new UnsupportedOperationException();
    }

    @Override
    public void reset() throws IOException {
        throw new UnsupportedOperationException();
    }

    @Override
    public boolean markSupported() {
        return false;
    }

    private long updateProgress(long numBytesRead) {
        if (numBytesRead > 0) {
            long oldTotalNumBytesRead = this.totalNumBytesRead;
            this.totalNumBytesRead += numBytesRead;
            propertyChangeSupport.firePropertyChange("totalNumBytesRead", oldTotalNumBytesRead, this.totalNumBytesRead);
        }

        return numBytesRead;
    }
}

Catchpole

Not bad. I probably would have forgotten skip(). :) – Zackzackariah

@David: To be honest implementing skip meant introducing nasty (int) casts so if you know you don't need it I'd throw an UnsupportedOperationException here too. – Catchpole

Also you need to check that super.read() don't return a negative (end of data). – Canthus

I think there's still a bug in there. The implementation of read(byte[]b) should just be return read(b, 0, b.length); (without a progress update). In the current implementation, calling read(byte[]b) causes the read bytes to be counted twice, because the implementation of FilterInputStream#read(byte[]b) directly calls read(byte[]b,int off,int len). – Hangup

Answer 2

Guava's com.google.common.io package can help you a little. The following is uncompiled and untested but should put you on the right track.

long total = file1.length();
long progress = 0;
final OutputStream out = new FileOutputStream(file2);
boolean success = false;
try {
  ByteStreams.readBytes(Files.newInputStreamSupplier(file1),
      new ByteProcessor<Void>() {
        public boolean processBytes(byte[] buffer, int offset, int length)
            throws IOException {
          out.write(buffer, offset, length);
          progress += length;
          updateProgressBar((double) progress / total);
          // or only update it periodically, if you prefer
        }
        public Void getResult() {
          return null;
        }
      });
  success = true;
} finally {
  Closeables.close(out, !success);
}

This may look like a lot of code, but I believe it's the least you'll get away with. (note other answers to this question don't give complete code examples, so it's hard to compare them that way.)

Heterocercal

Do you know of any reliable way to do this with a url and not a file ? What bugs me is the way to find the url's content length. As indicated here, there seems to be no way to know the size of the stream if we use gzip compression. android-developers.blogspot.fr/2011/09/… – Pleuron

(Shouldn't this comment be attached to that answer then?) – Heterocercal

Answer 3

The answer by Adamski works but there is a small bug. The overridden read(byte[] b) method calls the read(byte[] b, int off, int len) method trough the super class.
So updateProgress(long numBytesRead) is called twice for every read action and you end up with a numBytesRead that is twice the size of the file after the total file has been read.

Not overriding read(byte[] b) method solves the problem.

Bankston

This must be a comment of the @Catchpole answer not an answer by itself at least if you don't put the corrected code in your answer. – Canthus

Answer 4

If you’re building a GUI application there’s always ProgressMonitorInputStream. If there’s no GUI involved wrapping an InputStream in the way you describe is a no-brainer and takes less time than posting a question here.

Yell

Yeah, I looked at that. It's a GUI application but it's quite a complex one and progress reporting isn't just a matter of popping up a standard ProgressMonitor. Wrapping the InputStream would take less time than posting a question, but then I'd never know if someone out there has a better idea. – Zackzackariah

Answer 5

To complete the answer given by @Kevin Bourillion, it can be applied to a network content as well using this technique (that prevents reading the stream twice : one for size and one for content) :

        final HttpURLConnection httpURLConnection = (HttpURLConnection) new URL( url ).openConnection();
        InputSupplier< InputStream > supplier = new InputSupplier< InputStream >() {

            public InputStream getInput() throws IOException {
                return httpURLConnection.getInputStream();
            }
        };
        long total = httpURLConnection.getContentLength();
        final ByteArrayOutputStream bos = new ByteArrayOutputStream();
        ByteStreams.readBytes( supplier, new ProgressByteProcessor( bos, total ) );

Where ProgressByteProcessor is an inner class :

public class ProgressByteProcessor implements ByteProcessor< Void > {

    private OutputStream bos;
    private long progress;
    private long total;

    public ProgressByteProcessor( OutputStream bos, long total ) {
        this.bos = bos;
        this.total = total;
    }

    public boolean processBytes( byte[] buffer, int offset, int length ) throws IOException {
        bos.write( buffer, offset, length );
        progress += length - offset;
        publishProgress( (float) progress / total );
        return true;
    }

    public Void getResult() {
        return null;
    }
}

Pleuron

PS : This should not work with gzipped compressed urlconnections. – Pleuron