How to test for equality in ArrayBuffer, DataView, and TypedArray

Question

Is there a way how to test if two JavaScript ArrayBuffers are equal? I would like to write test for message composing method. The only way I found is to convert the ArrayBuffer to string and then compare. Did I miss something?

Following code is giving false, even if I think that it should be true:

(function() {
    'use strict';

    /* Fill buffer with data of Verse header and user_auth
     * command */
    var buf_pos = 0;
    var name_len = 6
    var message_len = 4 + 1 + 1 + 1 + name_len + 1;

    var buf = new ArrayBuffer(message_len);
    var view = new DataView(buf);
    /* Verse header starts with version */
    view.setUint8(buf_pos, 1 << 4); /* First 4 bits are reserved for version of protocol */
    buf_pos += 2;
    /* The lenght of the message */
    view.setUint16(buf_pos, message_len);
    buf_pos += 2;

    buf_pos = 0;
    var buf2 = new ArrayBuffer(message_len);
    var view2 = new DataView(buf);
    /* Verse header starts with version */
    view2.setUint8(buf_pos, 1 << 4); /* First 4 bits are reserved for version of protocol */
    buf_pos += 2;
    /* The lenght of the message */
    view2.setUint16(buf_pos, message_len);
    buf_pos += 2;


    if(buf == buf2){
        console.log('true');
    }
    else{
        console.log('false');
    }


}());

If I try to compare view and view2 it's false again.

Janniejanos

Answer 1

You cannot compare two objects directly in JavaScript using == or ===.
These operators will only check the equality of references (i.e. if expressions reference the same object).

You can, however, use DataView or ArrayView objects to retrieve values of specific parts of ArrayBuffer objects and check them.

If you want to check headers:

if (  view1.getUint8 (0) == view2.getUint8 (0)
   && view1.getUint16(2) == view2.getUint16(2)) ...

Or if you want to check the globality of your buffers:

function equal (buf1, buf2)
{
    if (buf1.byteLength != buf2.byteLength) return false;
    var dv1 = new Int8Array(buf1);
    var dv2 = new Int8Array(buf2);
    for (var i = 0 ; i != buf1.byteLength ; i++)
    {
        if (dv1[i] != dv2[i]) return false;
    }
    return true;
}

If you want to implement a complex data structure based on ArrayBuffer, I suggest creating your own class, or else you will have to resort to cumbersome raw DataView / ArrayView instances each time you will want to move a matchstick in and out of the structure.

Microphysics

Thanks. That's exactly what I need to do - implement set of classes for handling messages over websocket. I would like to have the methods and the state machine covered by tests. – Janniejanos

A guess ... will it work if we convert both arrays to String (say base 64) and then compare them – Marquetry

It could but I doubt that would be more efficient – Microphysics

Consider using DataView for better performance. See my answer below. – Marolda

@AnthumChris Why? The Int8Array is also just a "view" of the underlying buffer, it doesn't copy it. – Tifanie

I was rather thinking of the source code, which could become hard to read if you littered it with getUIntxxcalls. Also, maintaining consistency between your structure and the raw accesses might be easier if you put the whole handling in a neat module. TBH I have no idea how good JS has become at optimizing and whether one approach is faster than the other. – Microphysics

Answer 2

In general javascript, you currently have to compare two ArrayBuffer objects by wrapping each with a TypedArray, then manually iterating over each element and doing element-wise equality.

If the underlying buffer is 2 or 4-byte memory-aligned then you can make a significant optimization by employing Uint16 or Uint32 typed-arrays for the comparison.

/**
 * compare two binary arrays for equality
 * @param {(ArrayBuffer|ArrayBufferView)} a
 * @param {(ArrayBuffer|ArrayBufferView)} b 
 */
function equal(a, b) {
  if (a instanceof ArrayBuffer) a = new Uint8Array(a, 0);
  if (b instanceof ArrayBuffer) b = new Uint8Array(b, 0);
  if (a.byteLength != b.byteLength) return false;
  if (aligned32(a) && aligned32(b))
    return equal32(a, b);
  if (aligned16(a) && aligned16(b))
    return equal16(a, b);
  return equal8(a, b);
}

function equal8(a, b) {
  const ua = new Uint8Array(a.buffer, a.byteOffset, a.byteLength);
  const ub = new Uint8Array(b.buffer, b.byteOffset, b.byteLength);
  return compare(ua, ub);
}
function equal16(a, b) {
  const ua = new Uint16Array(a.buffer, a.byteOffset, a.byteLength / 2);
  const ub = new Uint16Array(b.buffer, b.byteOffset, b.byteLength / 2);
  return compare(ua, ub);
}
function equal32(a, b) {
  const ua = new Uint32Array(a.buffer, a.byteOffset, a.byteLength / 4);
  const ub = new Uint32Array(b.buffer, b.byteOffset, b.byteLength / 4);
  return compare(ua, ub);
}

function compare(a, b) {
  for (let i = a.length; -1 < i; i -= 1) {
    if ((a[i] !== b[i])) return false;
  }
  return true;
}

function aligned16(a) {
  return (a.byteOffset % 2 === 0) && (a.byteLength % 2 === 0);
}

function aligned32(a) {
  return (a.byteOffset % 4 === 0) && (a.byteLength % 4 === 0);
}

and called via:

equal(buf1, buf2)

here are the performance tests for 1-, 2-, 4-byte aligned memory.

Alternatives:

You may also get more performance with WASM, but its possible the cost of transferring the data to the heap may negate the comparison benefit.

Within Node.JS you may get more performance with Buffer as it will have native code: Buffer.from(buf1, 0).equals(Buffer.from(buf2, 0))

Resection

This code has two bugs: 1. If you passed a typed array that is not an instaneof Int8Array, Uint8Array, or Uint8ClampedArray, equal will not work correctly because .length will return the length in element units, not in bytes. You should use .byteLength instead. – Perplexity

2. Unfortunately, equal64 is useless no matter how fast it is. You can't use floating-point numbers to compare arbitrary buffers. It has both false positives and false negatives: equal64(new Uint8Array([0,0,0,0,0,0,0,0]), new Uint8Array([0,0,0,0,0,0,0,0x80])); // +0 and -0 → true equal64(new Uint8Array([0,0,0,0,0,0,0xF8,0xFF]), new Uint8Array([0,0,0,0,0,0,0xF8,0xFF])); // NaN → false – Perplexity

@Perplexity I did wonder about the float64 point, thanks for proving it. Thank you, I'll update and fix accordingly. – Resection

Note that BigUint64Array is now shipping. – Quoth

You can also further optimise the test by comparing an aligned prefix (e.g., given 131 bytes, comparing the first 131 bytes as longer words then comparing the final three bytes one-byte at a time). – Quoth

"Within Node.JS you may get more performance with Buffer as it will have native code: Buffer.from(buf1, 0).equals(Buffer.from(buf2, 0))" @MeirionHughes are there benchmarks online for this, to compare between the conversion to ArrayView and the native inbuilt Buffer.from ? Also I think we can skip the 2nd argument right as it will be 0 by default ? – Anabelanabella

I've never tested it, but you could setup a quick js to do it with npmjs.com/package/benchmark and post an answer. :) – Resection

Answer 3

To test for equality between two TypedArrays, consider using the every method, which exits as soon as an inconsistency is found:

const a = Uint8Array.from([0,1,2,3]);
const b = Uint8Array.from([0,1,2,3]);
const c = Uint8Array.from([0,1,2,3,4]);
const areEqual = (first, second) =>
    first.length === second.length && first.every((value, index) => value === second[index]);

console.log(areEqual(a, b));
console.log(areEqual(a, c));

This is less expensive than alternatives (like toString() comparisons) which iterate over the remaining array even after a difference is found.

Songwriter

Nice solution, but it needs a length check. This won't work if 'second' is longer than 'first', and the first bytes are equal. – Menjivar

Answer 4

In today's V8, DataView should now be "usable for performance-critical real-world applications" — https://v8.dev/blog/dataview

The functions below test equality based on the objects you already have instantiated. If you already have TypedArray objects, you could compare them directly without creating additional DataView objects for them (someone is welcome to measure performance for both options).

// compare ArrayBuffers
function arrayBuffersAreEqual(a, b) {
  return dataViewsAreEqual(new DataView(a), new DataView(b));
}

// compare DataViews
function dataViewsAreEqual(a, b) {
  if (a.byteLength !== b.byteLength) return false;
  for (let i=0; i < a.byteLength; i++) {
    if (a.getUint8(i) !== b.getUint8(i)) return false;
  }
  return true;
}

// compare TypedArrays
function typedArraysAreEqual(a, b) {
  if (a.byteLength !== b.byteLength) return false;
  return a.every((val, i) => val === b[i]);
}

Marolda

Answer 5

The IndexedDB API provides a built-in method for comparing two ArrayBuffers or ArrayBuffer views in a bytewise fashion. This is because IndexedDB has a concept of "key order", where a subset of JavaScript values that can be used as "keys" in IndexedDB have a defined sorting order. The important part is that one such kind of key is a binary key, which is defined in the spec as "ArrayBuffer objects (or views on buffers such as Uint8Array)."

Some caveats to this:

• Generally only available in the context of browsers. In NodeJS, you have Buffer-based options. I'm not sure about in other contexts.
• Only works if the IndexedDB API provided is v2.0 or above, because binary keys were not a feature in v1.0. According to caniuse.com, v2.0 is available in basically any browser that supports IndexedDB at all, which is the vast majority (except for IE, which apparently never even fully supported v1.0).
• Since comparison is bytewise, you can compare ArrayBuffers, Uint8Arrays and DataViews interchangeably. However, other kinds of typed array are a bit more complicated:
  • Generally, if both sides of the comparison are the same kind of typed array (i.e. both Int32Arrays, both Uint16Arrays, etc.), then you can at least test for equality using this method. But even then there is one place where it can give a surprising result: in float arrays, "negative" zero has a different bit-pattern than "positive" zero, so new Float64Array([0]) won't compare as equal to new Float64Array([-0]). This quirk does not apply to integer arrays.
  • Any other kind of comparison is likely to not give you the result you expect. For example, new Int8Array([-1]) will compare as greater than new Int8Array([0]). new Int32Array([1024]) will compare as less than new Int32Array([1]) on some machines but greater on others (!) due to typed arrays using native endianness. Float arrays will compare differing values as greater-than or less-than seemingly at random.
  • Basically, if you want to sort an array of typed arrays, this mechanism is only likely to help if they are all Uint8Arrays.
• It's just a bit weird

With all that being said, testing for binary equality with this method is simple enough:

function areBytewiseEqual(a, b) {
  return indexedDB.cmp(a, b) === 0;
}

Elka

Answer 6

I wrote these functions to compare the most normal data types. It works with ArrayBuffer, TypedArray, DataView, Node.js Buffer and any normal Array with byte data (0-255).

// It will not copy any underlying buffers, instead it will create a view into them.
function dataToUint8Array(data) {
  let uint8array
  if (data instanceof ArrayBuffer || Array.isArray(data)) {
    uint8array = new Uint8Array(data)
  } else if (data instanceof Buffer) { // Node.js Buffer
    uint8array = new Uint8Array(data.buffer, data.byteOffset, data.length)
  } else if (ArrayBuffer.isView(data)) { // DataView, TypedArray or Node.js Buffer
    uint8array = new Uint8Array(data.buffer, data.byteOffset, data.byteLength)
  } else {
    throw Error('Data is not an ArrayBuffer, TypedArray, DataView or a Node.js Buffer.')
  }
  return uint8array
}

function compareData(a, b) {
  a = dataToUint8Array(a); b = dataToUint8Array(b)
  if (a.byteLength != b.byteLength) return false
  return a.every((val, i) => val == b[i])
}

Tifanie

Answer 7

You can always convert the arrays into strings and compare them. E.g.

let a = new Uint8Array([1, 2, 3, 4]);
let b = new Uint8Array([1, 2, 3, 4]);
if (a.toString() == b.toString()) {
    console.log("Yes");
} else {
    console.log("No");
}

Demetriusdemeyer

This may not necessarily work if one of the Buffers is padded with 0 either side. If you toString a Buffer with padded 0's, the length of the returned string will not reflect the actual strings length, theres string comparisons will not work. e.g. Buffer.from([80]).toString() is "P". Buffer.from([80, 0]).toString() is also "P", but the length of the second is 2. Even though the string clearly only has one letter. Trimming the string doesn't seem to work either. – Allista

That would be the case if you were using node buffers. However, this answer uses the view - uint8array, calling the toString method returns the values joined together with ", ". Whereas your in your example, some string encoding method is used. Even then, the length difference is desirable as it prevents the collision. – Nilsanilsen