problem using base64 encoder and InputStreamReader

Question

I have some CLOB columns in a database that I need to put Base64 encoded binary files in. These files can be large, so I need to stream them, I can't read the whole thing in at once.

I'm using org.apache.commons.codec.binary.Base64InputStream to do the encoding, and I'm running into a problem. My code is essentially this

FileInputStream fis = new FileInputStream(file);
Base64InputStream b64is = new Base64InputStream(fis, true, -1, null);
BufferedReader reader = new BufferedReader(new InputStreamReader(b64is));

preparedStatement.setCharacterStream(1, reader);

When I run the above code, I get one of these during the execution of the update java.io.IOException: Underlying input stream returned zero bytes, it is thrown deep in the InputStreamReader code.

Why would this not work? It seems to me like the reader would attempt to read from the base 64 stream, which would read from the file stream, and everything should be happy.

Infighting

Answer 1

This appears to be a bug in Base64InputStream. You're calling it correctly.

You should report this to the Apache commons codec project.

Simple test case:

import java.io.*;
import org.apache.commons.codec.binary.Base64InputStream;

class tmp {
  public static void main(String[] args) throws IOException {
    FileInputStream fis = new FileInputStream(args[0]);
    Base64InputStream b64is = new Base64InputStream(fis, true, -1, null);

    while (true) {
      byte[] c = new byte[1024];
      int n = b64is.read(c);
      if (n < 0) break;
      if (n == 0) throw new IOException("returned 0!");
      for (int i = 0; i < n; i++) {
        System.out.print((char)c[i]);
      }
    }
  }
}

the read(byte[]) call of InputStream is not allowed to return 0. It does return 0 on any file which is a multiple of 3 bytes long.

Joubert

Yes, you're right. This is a bug in Base64InputStream. +1 for the testcase which confirms this. – Dionnadionne

Reported btw: issues.apache.org/jira/browse/CODEC-101 That said, I'm still wondering about the coincidence that my test file was indeed a multiple of 3 bytes long :o) – Dionnadionne

Wow, thanks for confirming that, I must say I'm surprised that I found such a bug (however inadvertently). – Infighting

Answer 2

Interesting, I did some tests here and it indeed throws that exception when you read the Base64InputStream using an InputStreamReader, regardless the source of the stream, but it works flawlessly when you read it as binary stream. As Trashgod mentioned, Base64 encoding is framed. The InputStreamReader should in fact have invoked flush() on the Base64InputStream once more to see if it doesn't return any more data.

I don't see other ways to fix this than implementing your own Base64InputStreamReader or Base64Reader. This is actually a bug, see Keith's answer.

As a workaround you can also just store it in a BLOB instead of a CLOB in the DB and use PreparedStatement#setBinaryStream() instead. It doesn't matter if it's stored as binary data or not. You don't want to have such large Base64 data to be indexable or searchable anyway.

---

Update: since that's not an option and having the Apache Commons Codec guys to fix the Base64InputStream bug which I repored as CODEC-101 might take some time, you may consider to use another 3rd party Base64 API. I've found one here (public domain, so you can do whatever with it you want, even place in your own package), I've tested it here and it works fine.

InputStream base64 = new Base64.InputStream(input, Base64.ENCODE);

---

Update 2: the commons codec guy has fixed it pretty soon.

Index: src/java/org/apache/commons/codec/binary/Base64InputStream.java
===================================================================
--- src/java/org/apache/commons/codec/binary/Base64InputStream.java (revision 950817)
+++ src/java/org/apache/commons/codec/binary/Base64InputStream.java (working copy)
@@ -145,21 +145,41 @@
         } else if (len == 0) {
             return 0;
         } else {
-            if (!base64.hasData()) {
-                byte[] buf = new byte[doEncode ? 4096 : 8192];
-                int c = in.read(buf);
-                // A little optimization to avoid System.arraycopy()
-                // when possible.
-                if (c > 0 && b.length == len) {
-                    base64.setInitialBuffer(b, offset, len);
+            int readLen = 0;
+            /*
+             Rationale for while-loop on (readLen == 0):
+             -----
+             Base64.readResults() usually returns > 0 or EOF (-1).  In the
+             rare case where it returns 0, we just keep trying.
+
+             This is essentially an undocumented contract for InputStream
+             implementors that want their code to work properly with
+             java.io.InputStreamReader, since the latter hates it when
+             InputStream.read(byte[]) returns a zero.  Unfortunately our
+             readResults() call must return 0 if a large amount of the data
+             being decoded was non-base64, so this while-loop enables proper
+             interop with InputStreamReader for that scenario.
+             -----
+             This is a fix for CODEC-101
+            */
+            while (readLen == 0) {
+                if (!base64.hasData()) {
+                    byte[] buf = new byte[doEncode ? 4096 : 8192];
+                    int c = in.read(buf);
+                    // A little optimization to avoid System.arraycopy()
+                    // when possible.
+                    if (c > 0 && b.length == len) {
+                        base64.setInitialBuffer(b, offset, len);
+                    }
+                    if (doEncode) {
+                        base64.encode(buf, 0, c);
+                    } else {
+                        base64.decode(buf, 0, c);
+                    }
                 }
-                if (doEncode) {
-                    base64.encode(buf, 0, c);
-                } else {
-                    base64.decode(buf, 0, c);
-                }
+                readLen = base64.readResults(b, offset, len);
             }
-            return base64.readResults(b, offset, len);
+            return readLen;
         }
     }

I tried it here and it works fine.

Dionnadionne

Unfortunately, I cannot use BLOB because sometimes the data in there will be text – Infighting

Answer 3

"For top efficiency, consider wrapping an InputStreamReader within a BufferedReader. For example:"

BufferedReader in = new BufferedReader(new InputStreamReader(b64is));

Addendum: As Base64 is padded to a multiple of 4 characters, verify that the source isn't truncated. A flush() may be required.

Waterway

Perhaps it is more efficient, but it doesn't solve the problem – Infighting

Any chance your stream is truncated? IIRC, base64 is framed. – Waterway

Question updated. Can you elaborate on what you mean by "base64 is framed" ? The stream comes directly from the file. – Infighting

The encoded stream must be padded to "an integer multiple of 4 characters" in order to decode the last byte; this would be a problem if the stream were truncated. Reference cited above. – Waterway

@Waterway - "Reference cited above.". WHERE? – Anoxemia

@Stephen C: "an integer multiple of 4 characters"—Base64 en.wikipedia.org/wiki/Base64 – Waterway