Looking to access 16-bit image data in Javascript/WebGL

Question

I'm trying to download 16-bit image data from a server and push it into a WebGL texture without browser plug-ins. texImage2d will work with: ImageData, HTMLImageElement, HTMLCanvasElement, or HTMLVideoElement. I'm looking for some javascript (a library or code sample) which can decode 16-bit TIFF or similar (hdf5, etc.) image data into one of these object types.

I have no problem doing this is 8-bit per channel RGB by using an to load a PNG but this doesn't work with 16-bit per channel data since there aren't any "standard" browser supported image formats which are 16-bit.

Pamalapamela

Answer 1

I don't think the main browsers natively support any 16-bit/channel image format at the moment.

One way to achieve the same effect would be to create two PNG images, one with the top 8 bits of each colour channel in the image and one with the bottom 8 bits. Then bind the images as two textures and combine the values in your shader, e.g.

highp float val = texture2d(samplerTop8bits, tex_coord) * (256.0 / 257.0);
val += texture2d(samplerBottom8bits, tex_coord) * (1.0 / 257.0);

(Note: you need highp precision to represent your data correctly in a 16-bit range)

Another method is only possible if floating point textures are supported in your target browser(s). You would, in the browser, combine the two PNG images into a floating point texture then access that normally. This may not be any faster and will probably use twice the amount of texture memory.

Cambist

I did something similar but used an RGB PNG and encoded the 2 bytes in the R and G channels then decoded in the shader. – Pamalapamela

Good suggestion @Trevor. Since I answered this question, I've done it in exactly that way. It's probably much more efficient than doing two texture reads. – Cambist

Can someone please give concrete example of this? Would appreciate that very much! – Giffie

Answer 2

In case of combining two PNG images, one with the top 8 bits and the second with the low 8 bits, I think it should be:

highp vec4 texCol = texture2D(tex_low, vec2(vTexCoord.s, vTexCoord.t)) * (1.0 / 257.0);
texCol += texture2D(tex_up, vec2(vTexCoord.s, vTexCoord.t)) * (256.0 / 257.0);

In 8 bits per channel RGB colors will range from 0 to 255 = 2^8 - 1.
In 16 bits per channel RGB colors will range from 0 to 65535 = 2^16 - 1 = 255*257.

Explanation

WebGL works using colour values from 0 to 1 and makes it by dividing 8 bit color value by 255. So the divided value belongs to the range <0,1>.
In case of 16 bit per channel we would like to divide it by 65535 to get the proper number from range <0,1>.

What we want is 16 bit color value reduced to range <0,1>.
Let low and up be color value from range 0..255. up is top 8 bits and low is low 8 bits.
To get 16 bit value we can compute: low + up*256. Now we have number in range 0..65535. To get value from range <0,1> we divide it by 65535. Note that WebGL works using color values from range <0,1> , it is Lw=low/255 and Uw=up/255. So, we don't have to multiply it by 255 and divide it by 65535 because 65535 = 255*257. Instead we just divide by 257.

Also I could not find any software to split 16 bit / channel image into two 8 bit/channel image, so here is my code, feel free to use it, it splits 16 bit / channel Tiff into two 8 bit/channel PNGs:

https://github.com/czero69/ImageSplitter

Barbitone

That is counter-intuitive, but correct! Definetely worth +1. – Kwh

Answer 3

PNGToy is a pretty featured library for extracting PNG chunks of almost all depths and channel modes with javascript (really client-side / without node.js, just Promise.js dependencies). The decode method will return the desired buffer. Here is an example for 16 bits grayscale PNG (16 bits RGB should work as well) :

var dataObj;
var img = new PngImage();
var buffer;

img.onload = function() {

    var pngtoy = this.pngtoy;

    dataObj = pngtoy.decode().then(function(results) {

        buffer = new Uint16Array(results.bitmap);

        for(var i = 0, j; i < buffer.length; i++) {

          j = buffer[i];
          buffer[i] = ((j & 0xff) << 8) | ((j & 0xff00) >>> 8); // needed to swap bytes for correct unsigned integer values  
        }

        console.log(buffer);
    });     
};

img.onerror = function(e) {
    console.log(e.message);
};

img.src = "image.png";

Automate

Looks like the links broken - try this instead github.com/neshume/pngtoy – Emetic

Note that neither github.com/neshume/pngtoy nor github.com/xgds/pngtoy have issues enabled, suggesting that neither is the official release, nor actually want to be open source projects, making both highly questionable as recommendation. – Scaife

Answer 4

Digging this up. This is how I use 16-bit floating point textures

    function createDataTexture16F(gl, width, height) {
      const texture = gl.createTexture()
      gl.bindTexture(gl.TEXTURE_2D, texture)
      gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE)
      gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE)
      gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR)
      gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR)
      gl.texStorage2D(gl.TEXTURE_2D, 0, gl.RGBA16F, width, height)
      return texture
    }

    function updateDataTexture16F(gl, texture, width, height, data32) {
      const data = float32ToHalfFloat(data32)
      gl.bindTexture(gl.TEXTURE_2D, texture)
      if (data.length !== width * height * 4) throw new Error('Incorrect data length')
      gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA16F, width, height, 0, gl.RGBA, gl.HALF_FLOAT, data)
    }

float32ToHalfFloat packs the data into Uint16Array as there's no Float16Array in JavaScript. data32 can be any array, either [] or Float32Array

    var toHalfFn = (function() {
      const floatView = new Float32Array(1)
      const int32View = new Int32Array(floatView.buffer)
      return function toHalf(val) {
        floatView[0] = val
        var x = int32View[0]
        var bits = (x >> 16) & 0x8000
        var m = (x >> 12) & 0x07ff
        var e = (x >> 23) & 0xff
        if (e < 103) {
          return bits
        }
        if (e > 142) {
          bits |= 0x7c00
          bits |= ((e == 255) ? 0 : 1) && (x & 0x007fffff)
          return bits
        }
        if (e < 113) {
          m |= 0x0800
          bits |= (m >> (114 - e)) + ((m >> (113 - e)) & 1)
          return bits
        }
        bits |= ((e - 112) << 10) | (m >> 1)
        bits += m & 1
        return bits
      }
    
    }())
    export const toHalf = toHalfFn
    
    function float32ToHalfFloat(arr) {
      const data = new Uint16Array(arr.length)
      for (var i = 0, il = arr.length; i < il; i ++) {
        data[i] = toHalf(arr[i])
      }
      return data
    }

Yirinec