Converting between strings and ArrayBuffers

Question

Is there a commonly accepted technique for efficiently converting JavaScript strings to ArrayBuffers and vice-versa? Specifically, I'd like to be able to write the contents of an ArrayBuffer to localStorage and then read it back.

Renfrew

I don't have any experience in this, but judging from the API documentation (khronos.org/registry/typedarray/specs/latest) if you build an Int8Array ArrayBufferView it might be possible to simply use the bracket notation to copy chars string[i] = buffer[i] and vice versa. – Miniskirt

@FK82, that looks like a reasonable approach (using Uint16Arrays for JS's 16-bit characters), but JavaScript strings are immutable so you can't assign directly to a character position. I would still need to copy String.fromCharCode(x) of each value in the Uint16Array to a normal Array and then call .join() on the Array. – Renfrew

@kpozin: True, didn't really think that through. – Miniskirt

@Renfrew It turns out that most modern JS engines have optimized string concatenation to the point where it's cheaper to just use string += String.fromCharCode(buffer[i]);. It does seem odd that there wouldn't be built-in methods for converting between strings and typed arrays. They had to know something like this would come up. – Seessel

arrayBuffer.toString() is working well for me. – Discophile

@citizen conn - I don't know what browser you're using, but on Chrome arrayBuffer.toString() just gives "[object ArrayBuffer]". Not very helpful. – Zumwalt

@download, Citation needed. Indeed, that isn't quite logically possible since the concatenation is done dynamically (as opposed to pre-compiled). Running += in a loop will give youj some serious bottlenecks. – Maroney

async version: str = await new Response(str).arrayBuffer() – Chiffonier

@Zumwalt Convert the ArrayBuffer to a TypedArray first. – Suckle

Answer 1

Update 2016 - five years on there are now new methods in the specs (see support below) to convert between strings and typed arrays using proper encoding.

TextEncoder

The TextEncoder represents:

The TextEncoder interface represents an encoder for a specific method, that is a specific character encoding, like utf-8, iso-8859-2, koi8, cp1261, gbk, ... An encoder takes a stream of code points as input and emits a stream of bytes.

Change note since the above was written: (ibid.)

Note: Firefox, Chrome and Opera used to have support for encoding types other than utf-8 (such as utf-16, iso-8859-2, koi8, cp1261, and gbk). As of Firefox 48 [...], Chrome 54 [...] and Opera 41, no other encoding types are available other than utf-8, in order to match the spec.*

*) Updated specs (W3) and here (whatwg).

After creating an instance of the TextEncoder it will take a string and encode it using a given encoding parameter:

if (!("TextEncoder" in window)) 
  alert("Sorry, this browser does not support TextEncoder...");

var enc = new TextEncoder(); // always utf-8
console.log(enc.encode("This is a string converted to a Uint8Array"));

You then of course use the .buffer parameter on the resulting Uint8Array to convert the underlaying ArrayBuffer to a different view if needed.

Just make sure that the characters in the string adhere to the encoding schema, for example, if you use characters outside the UTF-8 range in the example they will be encoded to two bytes instead of one.

For general use you would use UTF-16 encoding for things like localStorage.

TextDecoder

Likewise, the opposite process uses the TextDecoder:

The TextDecoder interface represents a decoder for a specific method, that is a specific character encoding, like utf-8, iso-8859-2, koi8, cp1261, gbk, ... A decoder takes a stream of bytes as input and emits a stream of code points.

All available decoding types can be found here.

if (!("TextDecoder" in window))
  alert("Sorry, this browser does not support TextDecoder...");

var enc = new TextDecoder("utf-8");
var arr = new Uint8Array([84,104,105,115,32,105,115,32,97,32,85,105,110,116,
                          56,65,114,114,97,121,32,99,111,110,118,101,114,116,
                          101,100,32,116,111,32,97,32,115,116,114,105,110,103]);
console.log(enc.decode(arr));

The MDN StringView library

An alternative to these is to use the StringView library (licensed as lgpl-3.0) which goal is:

• to create a C-like interface for strings (i.e., an array of character codes — an ArrayBufferView in JavaScript) based upon the JavaScript ArrayBuffer interface
• to create a highly extensible library that anyone can extend by adding methods to the object StringView.prototype
• to create a collection of methods for such string-like objects (since now: stringViews) which work strictly on arrays of numbers rather than on creating new immutable JavaScript strings
• to work with Unicode encodings other than JavaScript's default UTF-16 DOMStrings

giving much more flexibility. However, it would require us to link to or embed this library while TextEncoder/TextDecoder is being built-in in modern browsers.

Support

As of July/2018:

TextEncoder (Experimental, On Standard Track)

 Chrome    | Edge      | Firefox   | IE        | Opera     | Safari
 ----------|-----------|-----------|-----------|-----------|-----------
     38    |     ?     |    19°    |     -     |     25    |     -

 Chrome/A  | Edge/mob  | Firefox/A | Opera/A   |Safari/iOS | Webview/A
 ----------|-----------|-----------|-----------|-----------|-----------
     38    |     ?     |    19°    |     ?     |     -     |     38

°) 18: Firefox 18 implemented an earlier and slightly different version
of the specification.

WEB WORKER SUPPORT:

Experimental, On Standard Track

 Chrome    | Edge      | Firefox   | IE        | Opera     | Safari
 ----------|-----------|-----------|-----------|-----------|-----------
     38    |     ?     |     20    |     -     |     25    |     -

 Chrome/A  | Edge/mob  | Firefox/A | Opera/A   |Safari/iOS | Webview/A
 ----------|-----------|-----------|-----------|-----------|-----------
     38    |     ?     |     20    |     ?     |     -     |     38

Data from MDN - `npm i -g mdncomp` by epistemex

Rawboned

No support for TextDecoder from IE & Edge: caniuse.com/#search=TextDecoder – Yarborough

According to MS it's in development: developer.microsoft.com/en-us/microsoft-edge/platform/status/… – Publish

No support for Safari Mobile(ios) at 2018-04-18: developer.mozilla.org/en-US/docs/Web/API/TextDecoder – Obelia

One-liner: var encoder = 'TextEncoder' in window ? new TextEncoder() : {encode: function(str){return Uint8Array.from(str, function(c){return c.codePointAt(0);});}}; so you can just var array = encoder.encode('hello'); – Lantz

The thing with TextEncoder is that if you have binary data in a string (like, image), you don't want to use TextEncoder (apparently). Characters with code points greater than 127 produce two bytes. Why do I have binary data in a string? cy.fixture(NAME, 'binary') (cypress) produces a string. – Smoko

For IE support, you can use the FastestSmallestTextEncoderDecoder polyfill, as recommended by the MDN website. – Korikorie

You should wrap new TextDecoder("encoding") in a try/catch because you might have support for TextDecoder but not the character encoding you want to use. – Valueless

What does ibid mean? – Earhart

@Earhart ibid: "commonly used ... to refer to the source cited in the preceding note or list item." – Northwards

I don't think this should be the accepted answer. The question asked for "converting ArrayBuffer to string, and store in localStorage", and what this answer suggests is using TextDecoder to decode the binary data into string. However, we don't know whether the ArrayBuffer was created from a string in the first place (likely not). When it's not, from my test, it doesn't work as intended at all. (when converted from binary to string then back, it becomes completely different) – Imena

Answer 2

Although Dennis and gengkev solutions of using Blob/FileReader work, I wouldn't suggest taking that approach. It is an async approach to a simple problem, and it is much slower than a direct solution. I've made a post in html5rocks with a simpler and (much faster) solution: http://updates.html5rocks.com/2012/06/How-to-convert-ArrayBuffer-to-and-from-String

And the solution is:

function ab2str(buf) {
  return String.fromCharCode.apply(null, new Uint16Array(buf));
}

function str2ab(str) {
  var buf = new ArrayBuffer(str.length*2); // 2 bytes for each char
  var bufView = new Uint16Array(buf);
  for (var i=0, strLen=str.length; i<strLen; i++) {
    bufView[i] = str.charCodeAt(i);
  }
  return buf;
}

---

EDIT:

The Encoding API helps solving the string conversion problem. Check out the response from Jeff Posnik on Html5Rocks.com to the above original article.

Excerpt:

The Encoding API makes it simple to translate between raw bytes and native JavaScript strings, regardless of which of the many standard encodings you need to work with.

<pre id="results"></pre>

<script>
  if ('TextDecoder' in window) {
    // The local files to be fetched, mapped to the encoding that they're using.
    var filesToEncoding = {
      'utf8.bin': 'utf-8',
      'utf16le.bin': 'utf-16le',
      'macintosh.bin': 'macintosh'
    };

    Object.keys(filesToEncoding).forEach(function(file) {
      fetchAndDecode(file, filesToEncoding[file]);
    });
  } else {
    document.querySelector('#results').textContent = 'Your browser does not support the Encoding API.'
  }

  // Use XHR to fetch `file` and interpret its contents as being encoded with `encoding`.
  function fetchAndDecode(file, encoding) {
    var xhr = new XMLHttpRequest();
    xhr.open('GET', file);
    // Using 'arraybuffer' as the responseType ensures that the raw data is returned,
    // rather than letting XMLHttpRequest decode the data first.
    xhr.responseType = 'arraybuffer';
    xhr.onload = function() {
      if (this.status == 200) {
        // The decode() method takes a DataView as a parameter, which is a wrapper on top of the ArrayBuffer.
        var dataView = new DataView(this.response);
        // The TextDecoder interface is documented at http://encoding.spec.whatwg.org/#interface-textdecoder
        var decoder = new TextDecoder(encoding);
        var decodedString = decoder.decode(dataView);
        // Add the decoded file's text to the <pre> element on the page.
        document.querySelector('#results').textContent += decodedString + '\n';
      } else {
        console.error('Error while requesting', file, this);
      }
    };
    xhr.send();
  }
</script>

Conservator

I wrote a detailed answer on your html5rocks article (as soon it will be approved). – Braeunig

Will any arbitrary byte array survive a round trip with this method? Are there any circumstances under which we may lose data if a value in the array (e.g. 0xD800) happens to correspond to an invalid code point? The ultimate goal here is to have a reliable way to persist the contents of ArrayBuffers. – Renfrew

kpozin: in theory, it should. To make sure it does - at least in Chrome - you can run this code in the JavaScript console and make sure it doesn't throw an exception: for (var b=0x0000; b<=0xffff; b++) {abv[0]=b; if (new Uint16Array(str2ab(ab2str(ab)))[0]!==b) { throw "invalid conversion";} } – Conservator

@kpozin, one more thing: if you can change your implementation from localstorage to, for example, indexeddb, do it. Localstorage has several performance/concurrency problems that could get you in trouble under heavy usage - see this html5rocks article – Conservator

Unfortunately my comment on html5rocks is not approved yet. Therefore a short answer here. I still think, this isn't the right way, because you miss plenty of characters, especially because most pages are in UTF-8 encoding today. On one side, for more special characters (let's say Asian), the charCodeAt function returns a 4-Byte value, so they will be chopped. On the other side, simple English characters will grow the ArrayBuffer twice (you're using 2 Byte for every 1-Byte character). Imagine sending an English text over a WebSocket, it will need twice time (not good in real time environment). – Braeunig

@Dennis, I think this answer fits my use case. My main goal is to be able to persist the contents of ArrayBuffers that contain arbitrary binary data. I'm not (in this case) looking for an efficient conversion from strings containing actual text content. – Renfrew

@Dennis, JavaScript chars are 2-byte values. – Whack

Three examples: (1) This is a cool text! 20 Byte in UTF8 -- 40 Byte in Unicode. (2) ÄÖÜ 6 Bytes in UTF8 -- 6 Bytes in Unicode. (3) ☐☑☒ 9 Bytes in UTF8 -- 6 Bytes in Unicode. If you want to store the string as UTF8-file (via Blob and File Writer API), you cannot use this 2 methods, because the ArrayBuffer will be in Unicode and not in UTF8. – Braeunig

I get an error: Uncaught RangeError: Maximum call stack size exceeded. What could be the problem? – Schonfield

@Braeunig - JS strings use UCS2, not UTF8 (or even UTF16) - meaning charCodeAt() always returns values 0 -> 65535. Any UTF-8 code point that requires 4 bytes ends will be represented with surrogate pairs (see en.wikipedia.org/wiki/… ) - i.e. two separate 16-bit UCS2 values. – Lulu

@jacob - I believe the error is because there's a limit on the length of the array that can be passed to the apply() method. E.g. String.fromCharCode.apply(null, new Uint16Array(new ArrayBuffer(246300))).length works for me in Chrome, but if you use 246301 instead, I get your RangeError exception – Lulu

For even better performance, use TextDecoder/TextEncoder where available: developer.mozilla.org/en-US/docs/Web/API/TextDecoder – Protanopia

@broofa: charCodeAt() indeed always returns values less than 65536, but that does not mean that characters encoded by surrogates get 2 positions in the string: rather, it means we lose information when calling charCodeAt(), as it only returns the lower 16 bits of the real character code. – Fridafriday

@broofa: Sorry, a correction to my previous comments: it seems you're perfectly right; as unfortunate as it may be, JS strings are in fact not unicode strings :-/ – Fridafriday

When using ab2str: Uncaught RangeError: Maximum call stack size exceeded – Thresher

+1 for TextDecoder/TextEncoder: the html5rocks article now mentions it as a native, and probably faster, API... (in retrospect i'm not sure how i even came up with my original answer) – Deuce

If the solution doesn't work, or you get an unexpected set of characters, then try using one of the following TypedArray objects instead. In my case, the buf was a Uint8Array, so setting the return value of ab2str to return String.fromCharCode.apply(null, new Uint8Array(buf)); did the trick. – Favianus

Here's a version that handles all codepoints that JavaScript can handle: https://mcmap.net/q/20948/-conversion-between-utf-8-arraybuffer-and-string – Sophistry

Typed arrays are very slow input for String.fromCharCode.apply. It's much faster to append one character at a time in a loop - which also isn't limited by the call stack. – Veradia

@mangini,how it to be done if I have 32 bit array,please help. – Outrun

Good idea is to use Uint8Array, not 16. There may be problems with size if it's not dividable by 2. – Corrody

@Veradia If I'm not mistaken, building a string by repeated appending is a horrible O(n^2) situation due to strings being immutable – Verbal

@Steven Lu I am not familiar with that - could you give an example of a 'horrible O(n^2) situation'? In my use-case, I was only parsing a 12-character string. I forget what exactly I was doing, but Uint8Array didn't play well with fromCharCode.apply performance-wise. – Veradia

Well I certainly am not saying that fromCharCode.apply is any kind of a good idea either. Basically the point is that if your string is large, string = string + 'a' takes time linear in the length of string (because, not being mutable, the time taken to append any character cannot be amortized because a whole new string instance must be allocated). If strings are built via the repeated appending of single characters, this makes the entire string building process take quadratic time (with a linear number of allocations as opposed to a more palatable logarithmic number of them) – Verbal

This answer definitely should be removed, because it gives trash instead of data. – Allow

@Dennis: Javascript strings are more like an abomination mix of UCS-2 and UTF-16. Regardless of that, UCS-2, UTF-8, and UTF-16 are all implementations of unicode. JavaScript’s internal character encoding: UCS-2 or UTF-16? – Cambogia

One-liner for str2ab: function str2ab(str) { return (new Uint16Array(str.length)).map((_,i) => str.charCodeAt(i)); } – Telegraph

I get this error in Typescript for the ab2String function: Argument of type 'Uint16Array' is not assignable to parameter of type 'number[]'. Type 'Uint16Array' is missing the following properties from type 'number[]': pop, push, concat, shift, and 3 more. – Ardithardme

Answer 3

You can use TextEncoder and TextDecoder from the Encoding standard, which is polyfilled by the stringencoding library, to convert string to and from ArrayBuffers:

var uint8array = new TextEncoder().encode(string);
var string = new TextDecoder(encoding).decode(uint8array);

Paperweight

By the way, this is available in Firefox by default: developer.mozilla.org/en-US/docs/Web/API/TextDecoder.decode – Protanopia

Thumbs up for new APIs which are much better than weird workarounds! – Interlunar

This won't work with all types of characters out there. – Predestinarian

npm install text-encoding, var textEncoding = require('text-encoding'); var TextDecoder = textEncoding.TextDecoder;. No thanks. – Marino

grumble ... if I have an existing arraybuffer I want to write a string into I guess I have to take the uint8array and copy it a 2nd time?? – Lettielettish

I used this but was not getting a 1:1 (65 character string to 65 length array buffer). This works: new Uint8Array(str.split('').map((ea) => { return ea.charCodeAt(0) })); This was specifically for getting a private key into a Uint8Array for webpush. – Y

The fastest solution so far: jsbench.me/b6klaaxgwq/1 – Moidore

Answer 4

Blob is much slower than String.fromCharCode(null,array);

but that fails if the array buffer gets too big. The best solution I have found is to use String.fromCharCode(null,array); and split it up into operations that won't blow the stack, but are faster than a single char at a time.

The best solution for large array buffer is:

function arrayBufferToString(buffer){

    var bufView = new Uint16Array(buffer);
    var length = bufView.length;
    var result = '';
    var addition = Math.pow(2,16)-1;

    for(var i = 0;i<length;i+=addition){

        if(i + addition > length){
            addition = length - i;
        }
        result += String.fromCharCode.apply(null, bufView.subarray(i,i+addition));
    }

    return result;

}

I found this to be about 20 times faster than using blob. It also works for large strings of over 100mb.

Date

We should go with this solution. As this solves one more use case than the accepted one – Idler

I get: "Uncaught INVALID: json decode: this isn't json!" – Cayman

Answer 5

In case you have binary data in a string (obtained from nodejs + readFile(..., 'binary'), or cypress + cy.fixture(..., 'binary'), etc), you can't use TextEncoder. It supports only utf8. Bytes with values >= 128 are each turned into 2 bytes.

ES2015:

a = Uint8Array.from(s, x => x.charCodeAt(0))

Uint8Array(33) [2, 134, 140, 186, 82, 70, 108, 182, 233, 40, 143, 247, 29, 76, 245, 206, 29, 87, 48, 160, 78, 225, 242, 56, 236, 201, 80, 80, 152, 118, 92, 144, 48

s = String.fromCharCode.apply(null, a)

"ºRFl¶é(÷LõÎW0 Náò8ìÉPPv\0"

Crescent

Thank you for the simplest solution! – Inkhorn

Any way to make the fromCharCode TypeScript ready? It complains that a is not a number[] – Upi

This should be the accepted answer! – Sedimentary

That's the only answer that worked for me when using pdf.js. Thanks! – Retrorse

Answer 6

Based on the answer of gengkev, I created functions for both ways, because BlobBuilder can handle String and ArrayBuffer:

function string2ArrayBuffer(string, callback) {
    var bb = new BlobBuilder();
    bb.append(string);
    var f = new FileReader();
    f.onload = function(e) {
        callback(e.target.result);
    }
    f.readAsArrayBuffer(bb.getBlob());
}

and

function arrayBuffer2String(buf, callback) {
    var bb = new BlobBuilder();
    bb.append(buf);
    var f = new FileReader();
    f.onload = function(e) {
        callback(e.target.result)
    }
    f.readAsText(bb.getBlob());
}

A simple test:

string2ArrayBuffer("abc",
    function (buf) {
        var uInt8 = new Uint8Array(buf);
        console.log(uInt8); // Returns `Uint8Array { 0=97, 1=98, 2=99}`

        arrayBuffer2String(buf, 
            function (string) {
                console.log(string); // returns "abc"
            }
        )
    }
)

Braeunig

In arrayBuffer2String(), did you mean to call callback(...) instead of console.log()? Otherwise the callback argument goes unused. – Weakkneed

This looks like the way to go -- thanks genkev and Dennis. Seems kind of silly that there's no synchronous way to accomplish this, but what can you do... – Renfrew

JavaScript is single threaded. Therefore the FileReader is asynchronous for two reasons: (1) it will not block the execution of other JavaScript while loading a (huge) file (imagine a more complex application) and (2) it will not block the UI/Browser (common problem with long executing JS code). Plenty of APIs are asynchronous. Even in XMLHttpRequest 2 the synchronous is removed. – Braeunig

I was really hoping this would work for me, but the conversion from string to ArrayBuffer is not working reliably. I am making an ArrayBuffer with 256 values, and can turn that into a string with length 256. But then if I try to convert that back to an ArrayBuffer - depending on the contents of my initial ArrayBuffer - I'm getting out 376 elements. If you want to try to reproduce my problem, I'm treating my ArrayBuffer as a 16x16 grid in a Uint8Array, with values calculated as a[y * w + x] = (x + y) / 2 * 16; I've tried getBlob("x"), with many different mimetypes - no luck. – Len

BlobBuilder is deprecated in newer browsers. Change new BlobBuilder(); bb.append(buf); to new Blob([buf]), cast the ArrayBuffer in the second function to a UintArray via new UintArray(buf) (or whatever's appropriate for the underlying data type), and then get rid of the getBlob() calls. Finally, for cleanliness, rename bb to blob because it's not a BlobBuilder anymore. – Kyl

@Kyl do you have a reference for me? – Braeunig

developer.mozilla.org/en/DOM/…, and current tip-of-tree builds of Chrome give a log warning about deprecation. Also updates.html5rocks.com/2012/06/Don-t-Build-Blobs-Construct-Them – Kyl

I stumbled over the article on html5rocks by myself some days ago. It's in the W3C spec since 17th April: "The BlobBuilder interface is deprecated in favor of the new constructible Blob. However, at this time implementations generally support BlobBuilder and not constructible Blob." w3.org/TR/file-writer-api/#idl-def-BlobBuilder -- Thanks. – Braeunig

Answer 7

Just

const buffer = thisReturnsBuffers();

const blob = new Blob([buffer], {type: 'text/plain; charset=utf-8'});

blob.text().then(text => console.log(text));

Or

const stringVal = "string here";

const blob = new Blob([stringVal], {type: 'text/plain; charset=utf-8'});

blob.arrayBuffer().then(buffer => console.log(buffer));

Why are you all making this so complicated?

Bobbybobbye

I think this won't work in the case of string containing binary data? I think you'll need to put it in a uint8array first – Snipes

Ah, in the browser that would have already been stripped. Strings can't hold arbitrary bytes there. Is it different on Node? – Bobbybobbye

I'm not an expert but an example in the browser is the USVString you can receive from a blob. developer.mozilla.org/en-US/docs/Web/API/Blob/text – Snipes

If you're starting from a blob you probably shouldn't convert it to a string to then make a buffer, since blob.arrayBuffer is more direct. – Bobbybobbye

Answer 8

All the following is about getting binary strings from array buffers

I'd recommend not to use

var binaryString = String.fromCharCode.apply(null, new Uint8Array(arrayBuffer));

because it

1. crashes on big buffers (somebody wrote about "magic" size of 246300 but I got Maximum call stack size exceeded error on 120000 bytes buffer (Chrome 29))
2. it has really poor performance (see below)

If you exactly need synchronous solution use something like

var
  binaryString = '',
  bytes = new Uint8Array(arrayBuffer),
  length = bytes.length;
for (var i = 0; i < length; i++) {
  binaryString += String.fromCharCode(bytes[i]);
}

it is as slow as the previous one but works correctly. It seems that at the moment of writing this there is no quite fast synchronous solution for that problem (all libraries mentioned in this topic uses the same approach for their synchronous features).

But what I really recommend is using Blob + FileReader approach

function readBinaryStringFromArrayBuffer (arrayBuffer, onSuccess, onFail) {
  var reader = new FileReader();
  reader.onload = function (event) {
    onSuccess(event.target.result);
  };
  reader.onerror = function (event) {
    onFail(event.target.error);
  };
  reader.readAsBinaryString(new Blob([ arrayBuffer ],
    { type: 'application/octet-stream' }));
}

the only disadvantage (not for all) is that it is asynchronous. And it is about 8-10 times faster then previous solutions! (Some details: synchronous solution on my environment took 950-1050 ms for 2.4Mb buffer but solution with FileReader had times about 100-120 ms for the same amount of data. And I have tested both synchronous solutions on 100Kb buffer and they have taken almost the same time, so loop is not much slower the using 'apply'.)

BTW here: How to convert ArrayBuffer to and from String author compares two approaches like me and get completely opposite results (his test code is here) Why so different results? Probably because of his test string that is 1Kb long (he called it "veryLongStr"). My buffer was a really big JPEG image of size 2.4Mb.

Feinleib

Thanks for this async file reader solution. On 50MB chunks of a 717MB video file, it decreases the time to convert the buffer to a string by between 67% and 96%. That's huge. From many seconds to some hundreds of milliseconds – Oblique

Answer 9

(Update Please see the 2nd half of this answer, where I have (hopefully) provided a more complete solution.)

I also ran into this issue, the following works for me in FF 6 (for one direction):

var buf = new ArrayBuffer( 10 );
var view = new Uint8Array( buf );
view[ 3 ] = 4;
alert(Array.prototype.slice.call(view).join(""));

Unfortunately, of course, you end up with ASCII text representations of the values in the array, rather than characters. It still (should be) much more efficient than a loop, though. eg. For the example above, the result is 0004000000, rather than several null chars & a chr(4).

Edit:

After looking on MDC here, you may create an ArrayBuffer from an Array as follows:

var arr = new Array(23);
// New Uint8Array() converts the Array elements
//  to Uint8s & creates a new ArrayBuffer
//  to store them in & a corresponding view.
//  To get at the generated ArrayBuffer,
//  you can then access it as below, with the .buffer property
var buf = new Uint8Array( arr ).buffer;

To answer your original question, this allows you to convert ArrayBuffer <-> String as follows:

var buf, view, str;
buf = new ArrayBuffer( 256 );
view = new Uint8Array( buf );

view[ 0 ] = 7; // Some dummy values
view[ 2 ] = 4;

// ...

// 1. Buffer -> String (as byte array "list")
str = bufferToString(buf);
alert(str); // Alerts "7,0,4,..."

// 1. String (as byte array) -> Buffer    
buf = stringToBuffer(str);
alert(new Uint8Array( buf )[ 2 ]); // Alerts "4"

// Converts any ArrayBuffer to a string
//  (a comma-separated list of ASCII ordinals,
//  NOT a string of characters from the ordinals
//  in the buffer elements)
function bufferToString( buf ) {
    var view = new Uint8Array( buf );
    return Array.prototype.join.call(view, ",");
}
// Converts a comma-separated ASCII ordinal string list
//  back to an ArrayBuffer (see note for bufferToString())
function stringToBuffer( str ) {
    var arr = str.split(",")
      , view = new Uint8Array( arr );
    return view.buffer;
}

For convenience, here is a function for converting a raw Unicode String to an ArrayBuffer (will only work with ASCII/one-byte characters)

function rawStringToBuffer( str ) {
    var idx, len = str.length, arr = new Array( len );
    for ( idx = 0 ; idx < len ; ++idx ) {
        arr[ idx ] = str.charCodeAt(idx) & 0xFF;
    }
    // You may create an ArrayBuffer from a standard array (of values) as follows:
    return new Uint8Array( arr ).buffer;
}

// Alerts "97"
alert(new Uint8Array( rawStringToBuffer("abc") )[ 0 ]);

The above allow you to go from ArrayBuffer -> String & back to ArrayBuffer again, where the string may be stored in eg. .localStorage :)

Hope this helps,

Dan

Weakkneed

I don't think this is an efficient method (in terms of time or space), and this is a very unusual way to store binary data. – Renfrew

What about using base64 encoding? – Devitt

Answer 10

Unlike the solutions here, I needed to convert to/from UTF-8 data. For this purpose, I coded the following two functions, using the (un)escape/(en)decodeURIComponent trick. They're pretty wasteful of memory, allocating 9 times the length of the encoded utf8-string, though those should be recovered by gc. Just don't use them for 100mb text.

function utf8AbFromStr(str) {
    var strUtf8 = unescape(encodeURIComponent(str));
    var ab = new Uint8Array(strUtf8.length);
    for (var i = 0; i < strUtf8.length; i++) {
        ab[i] = strUtf8.charCodeAt(i);
    }
    return ab;
}

function strFromUtf8Ab(ab) {
    return decodeURIComponent(escape(String.fromCharCode.apply(null, ab)));
}

Checking that it works:

strFromUtf8Ab(utf8AbFromStr('latinкирилицаαβγδεζηあいうえお'))
-> "latinкирилицаαβγδεζηあいうえお"

Southeaster

A widely supported solution with a decent preformance: jsbench.me/b6klaaxgwq/1 – Moidore

Answer 11

For node.js and also for browsers using https://github.com/feross/buffer

function ab2str(buf: Uint8Array) {
  return Buffer.from(buf).toString('base64');
}
function str2ab(str: string) {
  return new Uint8Array(Buffer.from(str, 'base64'))
}

Note: Solutions here didn't work for me. I need to support node.js and browsers and just serialize UInt8Array to a string. I could serialize it as a number[] but that occupies unnecessary space. With that solution I don't need to worry about encodings since it's base64. Just in case other people struggle with the same problem... My two cents

Aggregate

Answer 12

I found I had problems with this approach, basically because I was trying to write the output to a file and it was non encoded properly. Since JS seems to use UCS-2 encoding (source, source), we need to stretch this solution a step further, here's my enhanced solution that works to me.

I had no difficulties with generic text, but when it was down to Arab or Korean, the output file didn't have all the chars but instead was showing error characters

File output: ","10k unit":"",Follow:"Õ©íüY‹","Follow %{screen_name}":"%{screen_name}U“’Õ©íü",Tweet:"ĤüÈ","Tweet %{hashtag}":"%{hashtag} ’ĤüÈY‹","Tweet to %{name}":"%{name}U“xĤüÈY‹"},ko:{"%{followers_count} followers":"%{followers_count}…X \Ì","100K+":"100Ì tÁ","10k unit":"Ì è",Follow:"\°","Follow %{screen_name}":"%{screen_name} Ø \°X0",K:"œ",M:"1Ì",Tweet:"¸","Tweet %{hashtag}":"%{hashtag}

Original: ","10k unit":"万",Follow:"フォローする","Follow %{screen_name}":"%{screen_name}さんをフォロー",Tweet:"ツイート","Tweet %{hashtag}":"%{hashtag} をツイートする","Tweet to %{name}":"%{name}さんへツイートする"},ko:{"%{followers_count} followers":"%{followers_count}명의 팔로워","100K+":"100만 이상","10k unit":"만 단위",Follow:"팔로우","Follow %{screen_name}":"%{screen_name} 님 팔로우하기",K:"천",M:"백만",Tweet:"트윗","Tweet %{hashtag}":"%{hashtag}

I took the information from dennis' solution and this post I found.

Here's my code:

function encode_utf8(s) {
  return unescape(encodeURIComponent(s));
}

function decode_utf8(s) {
  return decodeURIComponent(escape(s));
}

 function ab2str(buf) {
   var s = String.fromCharCode.apply(null, new Uint8Array(buf));
   return decode_utf8(decode_utf8(s))
 }

function str2ab(str) {
   var s = encode_utf8(str)
   var buf = new ArrayBuffer(s.length); 
   var bufView = new Uint8Array(buf);
   for (var i=0, strLen=s.length; i<strLen; i++) {
     bufView[i] = s.charCodeAt(i);
   }
   return bufView;
 }

This allows me to save the content to a file without encoding problems.

How it works: It basically takes the single 8-byte chunks composing a UTF-8 character and saves them as single characters (therefore an UTF-8 character built in this way, could be composed by 1-4 of these characters). UTF-8 encodes characters in a format that variates from 1 to 4 bytes in length. What we do here is encoding the sting in an URI component and then take this component and translate it in the corresponding 8 byte character. In this way we don't lose the information given by UTF8 characters that are more than 1 byte long.

Elburt

Answer 13

if you used huge array example arr.length=1000000 you can this code to avoid stack callback problems

function ab2str(buf) {
var bufView = new Uint16Array(buf);
var unis =""
for (var i = 0; i < bufView.length; i++) {
    unis=unis+String.fromCharCode(bufView[i]);
}
return unis
}

reverse function mangini answer from top

function str2ab(str) {
    var buf = new ArrayBuffer(str.length*2); // 2 bytes for each char
    var bufView = new Uint16Array(buf);
    for (var i=0, strLen=str.length; i<strLen; i++) {
        bufView[i] = str.charCodeAt(i);
    }
    return buf;
}

Platelayer

Answer 14

The following is a working Typescript implementation:

bufferToString(buffer: ArrayBuffer): string {
    return String.fromCharCode.apply(null, Array.from(new Uint16Array(buffer)));
}

stringToBuffer(value: string): ArrayBuffer {
    let buffer = new ArrayBuffer(value.length * 2); // 2 bytes per char
    let view = new Uint16Array(buffer);
    for (let i = 0, length = value.length; i < length; i++) {
        view[i] = value.charCodeAt(i);
    }
    return buffer;
}

I've used this for numerous operations while working with crypto.subtle.

Nocturn

This was my favorite approach, however, how could you do a check if "data" is Buffer? – Scamp

If the data is a string. If it is UTF16 or a smaller encoding, it can be turned into an ArrayBuffer with this function. It doesn't mean that buffer will be useful for other things, such as changing into an image. If you want to know that, you'll have to look at what the image format is and sniff it. Most file types have metadata at the start. – Nocturn

Answer 15

Well, here's a somewhat convoluted way of doing the same thing:

var string = "Blah blah blah", output;
var bb = new (window.BlobBuilder||window.WebKitBlobBuilder||window.MozBlobBuilder)();
bb.append(string);
var f = new FileReader();
f.onload = function(e) {
  // do whatever
  output = e.target.result;
}
f.readAsArrayBuffer(bb.getBlob());

Edit: BlobBuilder has long been deprecated in favor of the Blob constructor, which did not exist when I first wrote this post. Here's an updated version. (And yes, this has always been a very silly way to do the conversion, but it was just for fun!)

var string = "Blah blah blah", output;
var f = new FileReader();
f.onload = function(e) {
  // do whatever
  output = e.target.result;
};
f.readAsArrayBuffer(new Blob([string]));

Deuce

Answer 16

Recently I also need to do this for one of my project so did a well research and got a result from Google's Developer community which states this in a simple manner:

For ArrayBuffer to String

function ab2str(buf) {
  return String.fromCharCode.apply(null, new Uint16Array(buf));
}
// Here Uint16 can be different like Uinit8/Uint32 depending upon your buffer value type.

For String to ArrayBuffer

function str2ab(str) {
  var buf = new ArrayBuffer(str.length*2); // 2 bytes for each char
  var bufView = new Uint16Array(buf);
  for (var i=0, strLen=str.length; i < strLen; i++) {
    bufView[i] = str.charCodeAt(i);
  }
  return buf;
}
//Same here also for the Uint16Array.

For more in detail reference you can refer this blog by Google.

Bethannbethanne

This method inserted 'NUL' characters in my text… The solution that worked for me is https://mcmap.net/q/20884/-converting-between-strings-and-arraybuffers – Petticoat

Answer 17

  stringToArrayBuffer(byteString) {
    var byteArray = new Uint8Array(byteString.length);
    for (var i = 0; i < byteString.length; i++) {
      byteArray[i] = byteString.codePointAt(i);
    }
    return byteArray;
  }
  arrayBufferToString(buffer) {
    var byteArray = new Uint8Array(buffer);
    var byteString = '';
    for (var i = 0; i < byteArray.byteLength; i++) {
      byteString += String.fromCodePoint(byteArray[i]);
    }
    return byteString;
  }

Viable

this code is buggy if string contains unicode characters. example: arrayBufferToString(stringToArrayBuffer('🐴'))==='44' – Sungsungari

Answer 18

Let's say you have an arrayBuffer binaryStr:

let text = String.fromCharCode.apply(null, new Uint8Array(binaryStr));

and then you assign the text to the state.

Costanzia

Answer 19

See here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays/StringView (a C-like interface for strings based upon the JavaScript ArrayBuffer interface)

Tuna

That code is under the GPLv3. I think it's pretty unprofessional of Mozilla to even blend that code with their standards-compliant documentation. – Marcosmarcotte

Answer 20

After playing with mangini's solution for converting from ArrayBuffer to String - ab2str (which is the most elegant and useful one I have found - thanks!), I had some issues when handling large arrays. More specefivally, calling String.fromCharCode.apply(null, new Uint16Array(buf)); throws an error:

arguments array passed to Function.prototype.apply is too large.

In order to solve it (bypass) I have decided to handle the input ArrayBuffer in chunks. So the modified solution is:

function ab2str(buf) {
   var str = "";
   var ab = new Uint16Array(buf);
   var abLen = ab.length;
   var CHUNK_SIZE = Math.pow(2, 16);
   var offset, len, subab;
   for (offset = 0; offset < abLen; offset += CHUNK_SIZE) {
      len = Math.min(CHUNK_SIZE, abLen-offset);
      subab = ab.subarray(offset, offset+len);
      str += String.fromCharCode.apply(null, subab);
   }
   return str;
}

The chunk size is set to 2^16 because this was the size I have found to work in my development landscape. Setting a higher value caused the same error to reoccur. It can be altered by setting the CHUNK_SIZE variable to a different value. It is important to have an even number.

Note on performance - I did not make any performance tests for this solution. However, since it is based on the previous solution, and can handle large arrays, I see no reason why not to use it.

Osteoclasis

you can use typedarray.subarray to get a chunk at specified position and size, this is what i do to read headers off binary formats in js – Kersten

Answer 21

I used this and works for me.

function arrayBufferToBase64( buffer ) {
    var binary = '';
    var bytes = new Uint8Array( buffer );
    var len = bytes.byteLength;
    for (var i = 0; i < len; i++) {
        binary += String.fromCharCode( bytes[ i ] );
    }
    return window.btoa( binary );
}



function base64ToArrayBuffer(base64) {
    var binary_string =  window.atob(base64);
    var len = binary_string.length;
    var bytes = new Uint8Array( len );
    for (var i = 0; i < len; i++)        {
        bytes[i] = binary_string.charCodeAt(i);
    }
    return bytes.buffer;
}

Votyak

Answer 22

Yes:

const encstr = (`TextEncoder` in window) ? new TextEncoder().encode(str) : Uint8Array.from(str, c => c.codePointAt(0));

Antiproton

Answer 23

For me this worked well.

  static async hash(message) {
    const data = new TextEncoder().encode(message);
    const hashBuffer = await crypto.subtle.digest('SHA-256', data)
    const hashArray = Array.from(new Uint8Array(hashBuffer))
    const hashHex = hashArray.map((b) => b.toString(16).padStart(2, '0')).join('')
    return hashHex
  }

Entranceway

Answer 24

The "native" binary string that atob() returns is a 1-byte-per-character Array.

So we shouldn't store 2 byte into a character.

var arrayBufferToString = function(buffer) {
  return String.fromCharCode.apply(null, new Uint8Array(buffer));
}

var stringToArrayBuffer = function(str) {
  return (new Uint8Array([].map.call(str,function(x){return x.charCodeAt(0)}))).buffer;
}

Selfrestraint

Answer 25

ArrayBuffer -> Buffer -> String(Base64)

Change ArrayBuffer to Buffer and then to String.

Buffer.from(arrBuffer).toString("base64");

Yammer

Any way to convert back? – Arenicolous

@MaximillianLaumeister Yes, see cancerbero's answer: https://mcmap.net/q/20884/-converting-between-strings-and-arraybuffers. Surprised how elegant this gets amidst all the hairy solutions here! – Metabolite

This is node-only solution, does not work in browsers. – Kaceykachina

Answer 26

Use splat unpacking instead of loops:

arrbuf = new Uint8Array([104, 101, 108, 108, 111])
text = String.fromCharCode(...arrbuf)
console.log(text)

For substrings arrbuf.slice() can be employed.

Belomancy

It won't work for non ascii charcters (utf-8 bytes) – Arnelle

Answer 27

I'd recommend NOT using deprecated APIs like BlobBuilder

BlobBuilder has long been deprecated by the Blob object. Compare the code in Dennis' answer — where BlobBuilder is used — with the code below:

function arrayBufferGen(str, cb) {

  var b = new Blob([str]);
  var f = new FileReader();

  f.onload = function(e) {
    cb(e.target.result);
  }

  f.readAsArrayBuffer(b);

}

Note how much cleaner and less bloated this is compared to the deprecated method... Yeah, this is definitely something to consider here.

Variscite

I mean, yes, but that Blob constructor wasn't really usable back in 2012 ;) – Deuce

Answer 28

var decoder = new TextDecoder ();
var string = decoder.decode (arrayBuffer);

See https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder/decode

Reversion

Answer 29

From emscripten:

function stringToUTF8Array(str, outU8Array, outIdx, maxBytesToWrite) {
  if (!(maxBytesToWrite > 0)) return 0;
  var startIdx = outIdx;
  var endIdx = outIdx + maxBytesToWrite - 1;
  for (var i = 0; i < str.length; ++i) {
    var u = str.charCodeAt(i);
    if (u >= 55296 && u <= 57343) {
      var u1 = str.charCodeAt(++i);
      u = 65536 + ((u & 1023) << 10) | u1 & 1023
    }
    if (u <= 127) {
      if (outIdx >= endIdx) break;
      outU8Array[outIdx++] = u
    } else if (u <= 2047) {
      if (outIdx + 1 >= endIdx) break;
      outU8Array[outIdx++] = 192 | u >> 6;
      outU8Array[outIdx++] = 128 | u & 63
    } else if (u <= 65535) {
      if (outIdx + 2 >= endIdx) break;
      outU8Array[outIdx++] = 224 | u >> 12;
      outU8Array[outIdx++] = 128 | u >> 6 & 63;
      outU8Array[outIdx++] = 128 | u & 63
    } else {
      if (outIdx + 3 >= endIdx) break;
      outU8Array[outIdx++] = 240 | u >> 18;
      outU8Array[outIdx++] = 128 | u >> 12 & 63;
      outU8Array[outIdx++] = 128 | u >> 6 & 63;
      outU8Array[outIdx++] = 128 | u & 63
    }
  }
  outU8Array[outIdx] = 0;
  return outIdx - startIdx
}

Use like:

stringToUTF8Array('abs', new Uint8Array(3), 0, 4);

Rosas