how do I access XHR responseBody (for binary data) from Javascript in IE?

Question

I've got a web page that uses XMLHttpRequest to download a binary resource.

In Firefox and Gecko I can use responseText to get the bytes, even if the bytestream includes binary zeroes. I may need to coerce the mimetype with overrideMimeType() to make that happen. In IE, though, responseText doesn't work, because it appears to terminate at the first zero. If you read 100,000 bytes, and byte 7 is a binary zero, you will be able to access only 7 bytes. IE's XMLHttpRequest exposes a responseBody property to access the bytes. I've seen a few posts suggesting that it's impossible to access this property in any meaningful way directly from Javascript. This sounds crazy to me.

xhr.responseBody is accessible from VBScript, so the obvious workaround is to define a method in VBScript in the webpage, and then call that method from Javascript. See jsdap for one example. EDIT: DO NOT USE THIS VBScript!!

var IE_HACK = (/msie/i.test(navigator.userAgent) && 
               !/opera/i.test(navigator.userAgent));   

// no no no!  Don't do this! 
if (IE_HACK) document.write('<script type="text/vbscript">\n\
     Function BinaryToArray(Binary)\n\
         Dim i\n\
         ReDim byteArray(LenB(Binary))\n\
         For i = 1 To LenB(Binary)\n\
             byteArray(i-1) = AscB(MidB(Binary, i, 1))\n\
         Next\n\
         BinaryToArray = byteArray\n\
     End Function\n\
</script>'); 

var xml = (window.XMLHttpRequest) 
    ? new XMLHttpRequest()      // Mozilla/Safari/IE7+
    : (window.ActiveXObject) 
      ? new ActiveXObject("MSXML2.XMLHTTP")  // IE6
      : null;  // Commodore 64?


xml.open("GET", url, true);
if (xml.overrideMimeType) {
    xml.overrideMimeType('text/plain; charset=x-user-defined');
} else {
    xml.setRequestHeader('Accept-Charset', 'x-user-defined');
}

xml.onreadystatechange = function() {
    if (xml.readyState == 4) {
        if (!binary) {
            callback(xml.responseText);
        } else if (IE_HACK) {
            // call a VBScript method to copy every single byte
            callback(BinaryToArray(xml.responseBody).toArray());
        } else {
            callback(getBuffer(xml.responseText));
        }
    }
};
xml.send('');

Is this really true? The best way? copying every byte? For a large binary stream that's not going to be very efficient.

There is also a possible technique using ADODB.Stream, which is a COM equivalent of a MemoryStream. See here for an example. It does not require VBScript but does require a separate COM object.

if (typeof (ActiveXObject) != "undefined" && typeof (httpRequest.responseBody) != "undefined") {
    // Convert httpRequest.responseBody byte stream to shift_jis encoded string
    var stream = new ActiveXObject("ADODB.Stream");
    stream.Type = 1; // adTypeBinary
    stream.Open ();
    stream.Write (httpRequest.responseBody);
    stream.Position = 0;
    stream.Type = 1; // adTypeBinary;
    stream.Read....          /// ???? what here
}

But that's not going to work well - ADODB.Stream is disabled on most machines these days.

---

In The IE8 developer tools - the IE equivalent of Firebug - I can see the responseBody is an array of bytes and I can even see the bytes themselves. The data is right there. I don't understand why I can't get to it.

Is it possible for me to read it with responseText?

hints? (other than defining a VBScript method)

Pneuma

Answer 1

Yes, the answer I came up with for reading binary data via XHR in IE, is to use VBScript injection. This was distasteful to me at first, but, I look at it as just one more browser dependent bit of code. (The regular XHR and responseText works fine in other browsers; you may have to coerce the mime type with XMLHttpRequest.overrideMimeType(). This isn't available on IE).

This is how I got a thing that works like responseText in IE, even for binary data. First, inject some VBScript as a one-time thing, like this:

if(/msie/i.test(navigator.userAgent) && !/opera/i.test(navigator.userAgent)) {
    var IEBinaryToArray_ByteStr_Script =
    "<!-- IEBinaryToArray_ByteStr -->\r\n"+
    "<script type='text/vbscript' language='VBScript'>\r\n"+
    "Function IEBinaryToArray_ByteStr(Binary)\r\n"+
    "   IEBinaryToArray_ByteStr = CStr(Binary)\r\n"+
    "End Function\r\n"+
    "Function IEBinaryToArray_ByteStr_Last(Binary)\r\n"+
    "   Dim lastIndex\r\n"+
    "   lastIndex = LenB(Binary)\r\n"+
    "   if lastIndex mod 2 Then\r\n"+
    "       IEBinaryToArray_ByteStr_Last = Chr( AscB( MidB( Binary, lastIndex, 1 ) ) )\r\n"+
    "   Else\r\n"+
    "       IEBinaryToArray_ByteStr_Last = "+'""'+"\r\n"+
    "   End If\r\n"+
    "End Function\r\n"+
    "</script>\r\n";

    // inject VBScript
    document.write(IEBinaryToArray_ByteStr_Script);
}

The JS class I'm using that reads binary files exposes a single interesting method, readCharAt(i), which reads the character (a byte, really) at the i'th index. This is how I set it up:

// see doc on http://msdn.microsoft.com/en-us/library/ms535874(VS.85).aspx
function getXMLHttpRequest() 
{
    if (window.XMLHttpRequest) {
        return new window.XMLHttpRequest;
    }
    else {
        try {
            return new ActiveXObject("MSXML2.XMLHTTP"); 
        }
        catch(ex) {
            return null;
        }
    }
}

// this fn is invoked if IE
function IeBinFileReaderImpl(fileURL){
    this.req = getXMLHttpRequest();
    this.req.open("GET", fileURL, true);
    this.req.setRequestHeader("Accept-Charset", "x-user-defined");
    // my helper to convert from responseBody to a "responseText" like thing
    var convertResponseBodyToText = function (binary) {
        var byteMapping = {};
        for ( var i = 0; i < 256; i++ ) {
            for ( var j = 0; j < 256; j++ ) {
                byteMapping[ String.fromCharCode( i + j * 256 ) ] =
                    String.fromCharCode(i) + String.fromCharCode(j);
            }
        }
        // call into VBScript utility fns
        var rawBytes = IEBinaryToArray_ByteStr(binary);
        var lastChr = IEBinaryToArray_ByteStr_Last(binary);
        return rawBytes.replace(/[\s\S]/g,
                                function( match ) { return byteMapping[match]; }) + lastChr;
    };

    this.req.onreadystatechange = function(event){
        if (that.req.readyState == 4) {
            that.status = "Status: " + that.req.status;
            //that.httpStatus = that.req.status;
            if (that.req.status == 200) {
                // this doesn't work
                //fileContents = that.req.responseBody.toArray(); 

                // this doesn't work
                //fileContents = new VBArray(that.req.responseBody).toArray(); 

                // this works...
                var fileContents = convertResponseBodyToText(that.req.responseBody);

                fileSize = fileContents.length-1;
                if(that.fileSize < 0) throwException(_exception.FileLoadFailed);
                that.readByteAt = function(i){
                    return fileContents.charCodeAt(i) & 0xff;
                };
            }
            if (typeof callback == "function"){ callback(that);}
        }
    };
    this.req.send();
}

// this fn is invoked if non IE
function NormalBinFileReaderImpl(fileURL){
    this.req = new XMLHttpRequest();
    this.req.open('GET', fileURL, true);
    this.req.onreadystatechange = function(aEvt) {
        if (that.req.readyState == 4) {
            if(that.req.status == 200){
                var fileContents = that.req.responseText;
                fileSize = fileContents.length;

                that.readByteAt = function(i){
                    return fileContents.charCodeAt(i) & 0xff;
                }
                if (typeof callback == "function"){ callback(that);}
            }
            else
                throwException(_exception.FileLoadFailed);
        }
    };
    //XHR binary charset opt by Marcus Granado 2006 [http://mgran.blogspot.com] 
    this.req.overrideMimeType('text/plain; charset=x-user-defined');
    this.req.send(null);
}

The conversion code was provided by Miskun.

Very fast, works great.

I used this method to read and extract zip files from Javascript, and also in a class that reads and displays EPUB files in Javascript. Very reasonable performance. About half a second for a 500kb file.

Pneuma

What did you use to read binary data in other browsers? I see your question references a getBuffer function. What is that? AFAIK only IE supports responseBody. So what did you use ?? – Outoftheway

I used MSXML2.XMLHTTP in IE, and XMLHttpRequest() in non-IE. In the non-IE browsers, I was able to use responseText to get the byte stream. Apparently IE thinks it is a string, and therefore in IE, beyond the first zero, responseText[i] returns "undefined". But that is not so for FF3.5. It just works. – Pneuma

But...then it may be a different issue: IE does not allow the [] operator on strings. It uses .charAt(num) instead. So it may just be you need responseText.charAt(i)? – Outoftheway

I used responseText.charCodeAt(i) - it didn't work in IE, after the first zero. I just updated the post with more code, to illustrate. – Pneuma

Thank you. Now I need to take a bath... I feel so dirty. But so glad this fixed my issue. – Facial

Answer 2

XMLHttpRequest.responseBody is a VBArray object containing the raw bytes. You can convert these objects to standard arrays using the toArray() function:

var data = xhr.responseBody.toArray();

Loyce

I believe I tried that and it did not work; can't recall now what the problem was, though. – Pneuma

Do give it another try, I've just confirmed it works as expected for me. Should just be a drop in replacement where you're doing: var fileContents = BinaryToArray(that.req.responseBody).toArray(); – Loyce

Should also consider doing feature detection instead of looking at the user agent string. Just check if "overridemimetype" exists, if not, check if VBArray exists, if not, punt ;) – Loyce

@timrice: "VBArray" method doesn't work with IE8 or earlier. It will throw "VBArray expected" TypeError. Feature detection does not work either because "VBArray" object itself exists on IE8. Please stop spreading the flawed method blindly. Although it works with IE9 and later (even in IE8 mode), it is not so faster than "VBScript+CStr" method. – Furious

Thanks, works great for me in IE9 (my minimum supported IE version anyway), and is a much simpler solution. – Opal

Answer 3

I would suggest two other (fast) options:

1. First, you can use ADODB.Recordset to convert the byte array into a string. I would guess that this object is more common that ADODB.Stream, which is often disabled for security reasons. This option is VERY fast, less than 30ms for a 500kB file.

2. Second, if the Recordset component is not accessible, there is a trick to access the byte array data from Javascript. Send your xhr.responseBody to VBScript, pass it through any VBScript string function such as CStr (takes no time), and return it to JS. You will get a weird string with bytes concatenated into 16-bit unicode (in reverse). You can then convert this string quickly into a usable bytestring through a regular expression with dictionary-based replacement. Takes about 1s for 500kB.

For comparison, the byte-by-byte conversion through loops takes several minutes for this same 500kB file, so it's a no-brainer :) Below the code I have been using, to insert into your header. Then call the function ieGetBytes with your xhr.responseBody.

<!--[if IE]>    
<script type="text/vbscript">

    'Best case scenario when the ADODB.Recordset object exists
    'We will do the existence test in Javascript (see after)
    'Extremely fast, about 25ms for a 500kB file
    Function ieGetBytesADO(byteArray)
        Dim recordset
        Set recordset = CreateObject("ADODB.Recordset")
        With recordset
            .Fields.Append "temp", 201, LenB(byteArray)
            .Open
            .AddNew
            .Fields("temp").AppendChunk byteArray
            .Update
        End With
        ieGetBytesADO = recordset("temp")
        recordset.Close
        Set recordset = Nothing
    End Function

    'Trick to return a Javascript-readable string from a VBScript byte array
    'Yet the string is not usable as such by Javascript, since the bytes
    'are merged into 16-bit unicode characters. Last character missing if odd length.
    Function ieRawBytes(byteArray)
        ieRawBytes = CStr(byteArray)
    End Function

    'Careful the last character is missing in case of odd file length
    'We Will call the ieLastByte function (below) from Javascript
    'Cannot merge directly within ieRawBytes as the final byte would be duplicated
    Function ieLastChr(byteArray)
        Dim lastIndex
        lastIndex = LenB(byteArray)
        if lastIndex mod 2 Then
            ieLastChr = Chr( AscB( MidB( byteArray, lastIndex, 1 ) ) )
        Else
            ieLastChr = ""
        End If
    End Function

</script>

<script type="text/javascript">
    try {   
        // best case scenario, the ADODB.Recordset object exists
        // we can use the VBScript ieGetBytes function to transform a byte array into a string
        var ieRecordset = new ActiveXObject('ADODB.Recordset');
        var ieGetBytes = function( byteArray ) {
            return ieGetBytesADO(byteArray);
        }
        ieRecordset = null;

    } catch(err) {
        // no ADODB.Recordset object, we will do the conversion quickly through a regular expression

        // initializes for once and for all the translation dictionary to speed up our regexp replacement function
        var ieByteMapping = {};
        for ( var i = 0; i < 256; i++ ) {
            for ( var j = 0; j < 256; j++ ) {
                ieByteMapping[ String.fromCharCode( i + j * 256 ) ] = String.fromCharCode(i) + String.fromCharCode(j);
            }
        }

        // since ADODB is not there, we replace the previous VBScript ieGetBytesADO function with a regExp-based function,
        // quite fast, about 1.3 seconds for 500kB (versus several minutes for byte-by-byte loops over the byte array)
        var ieGetBytes = function( byteArray ) {
            var rawBytes = ieRawBytes(byteArray),
                lastChr = ieLastChr(byteArray);

            return rawBytes.replace(/[\s\S]/g, function( match ) {
                return ieByteMapping[match]; }) + lastChr;
        }
    }
</script>
<![endif]-->

Pullman

The ADODB method doesn't work. Fast as it might be, it returns characters are that beyond the standard ASCII range. Thus, your data ends up corrupted. Now, if there was a neat way to convert from an "Array of Bytes" straight to a JS Array (without using VBArray), that would be faster. – Evergreen

Does the "byte-by-byte conversion through loops" correspond to this answer? – Lava

Answer 4

Thanks so much for this solution. the BinaryToArray() function in VbScript works great for me.

Incidentally, I need the binary data for providing it to an Applet. (Don't ask me why Applets can't be used for downloading binary data. Long story short.. weird MS authentication that cant go thru applets (URLConn) calls. Its especially weird in cases where users are behind a proxy )

The Applet needs a byte array from this data, so here's what I do to get it:

 String[] results = result.toString().split(",");
    byte[] byteResults = new byte[results.length];
    for (int i=0; i<results.length; i++){
        byteResults[i] = (byte)Integer.parseInt(results[i]);
    }

The byte array can then converted into a bytearrayinputstream for further processing.

Yate

Answer 5

You could also just make a proxy script that goes to the address you're requesting & base64's it. Then you just have to pass a query string to the proxy script that tells it the address. In IE you have to manually do base64 in JS though. But this is a way to go if you don't want to use VBScript.

I used this for my GameBoy Color emulator.

Here is the PHP script that does the magic:

<?php
//Binary Proxy
if (isset($_GET['url'])) {
    try {
        $curl = curl_init();
        curl_setopt($curl, CURLOPT_URL, stripslashes($_GET['url']));
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($curl, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
        curl_setopt($curl, CURLOPT_POST, false);
        curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 30);
        $result = curl_exec($curl);
        curl_close($curl);
        if ($result !== false) {
            header('Content-Type: text/plain; charset=ASCII');
            header('Expires: '.gmdate('D, d M Y H:i:s \G\M\T', time() + (3600 * 24 * 7)));
            echo(base64_encode($result));
        }
        else {
            header('HTTP/1.0 404 File Not Found');
        }
    }
    catch (Exception $error) { }
}
?>

Utrillo

Answer 6

Thank you for this post.

I found this link usefull:

http://www.codingforums.com/javascript-programming/47018-help-using-responsetext-property-microsofts-xmlhttp-activexobject-ie6.html

Specially this part:

</script>
<script language="VBScript">
Function BinaryToString(Binary)
Dim I,S
For I = 1 to LenB(Binary)
S = S & Chr(AscB(MidB(Binary,I,1)))
Next
BinaryToString = S
End Function
</script>

I've added this to my htm page. Then I call this function from my javascript:

 responseText = BinaryToString(xhr.responseBody);

Works on IE8, IE9, IE10, FF & Chrome.

Jurisprudent

Answer 7

I was trying to download a file and than sign it using CAPICOM.DLL. The only way I coud do it was by injecting a VBScript function that does the download. That is my solution:

if(/msie/i.test(navigator.userAgent) && !/opera/i.test(navigator.userAgent)) {
    var VBConteudo_Script =
    '<!-- VBConteudo -->\r\n'+
    '<script type="text/vbscript">\r\n'+
    'Function VBConteudo(url)\r\n'+
    '   Set objHTTP = CreateObject("MSXML2.XMLHTTP")\r\n'+
    '   objHTTP.open "GET", url, False\r\n'+
    '   objHTTP.send\r\n'+
    '   If objHTTP.Status = 200 Then\r\n'+
    '       VBConteudo = objHTTP.responseBody\r\n'+
    '   End If\r\n'+
    'End Function\r\n'+
    '\<\/script>\r\n';

    // inject VBScript
    document.write(VBConteudo_Script);
}

Hellenistic