How do I ungzip (decompress) a NodeJS request's module gzip response body?

Question

How do I unzip a gzipped body in a request's module response?

I have tried several examples around the web but none of them appear to work.

request(url, function(err, response, body) {
    if(err) {
        handleError(err)
    } else {
        if(response.headers['content-encoding'] == 'gzip') {    
            // How can I unzip the gzipped string body variable?
            // For instance, this url:
            // http://highsnobiety.com/2012/08/25/norse-projects-fall-2012-lookbook/
            // Throws error:
            // { [Error: incorrect header check] errno: -3, code: 'Z_DATA_ERROR' }
            // Yet, browser displays page fine and debugger shows its gzipped
            // And unzipped by browser fine...
            if(response.headers['content-encoding'] && response.headers['content-encoding'].toLowerCase().indexOf('gzip') > -1) {   
                var body = response.body;                    
                zlib.gunzip(response.body, function(error, data) {
                    if(!error) {
                        response.body = data.toString();
                    } else {
                        console.log('Error unzipping:');
                        console.log(error);
                        response.body = body;
                    }
                });
            }
        }
    }
}

Argentinaargentine

Shouldn't the browser transparently do that? – Hosfmann

I added the node.js tag, but I get that does not make it clear... let me edit the post... – Argentinaargentine

can you save data to file req.gz and unzip it from command line? If yes, what is the output from gunzip req.gz and file req.gz – Saguaro

Hi Andrew! Thanks for the suggestion. If I save the file to a "req.gz" file, extracting it on the desktop produces a file named "req.gz.cpgz". Extacting this file in turn produces a 3rd file namded "req 2.gz". The request body was encoded to utf8 (response.setEncoding('utf8')) prior to reading the body. However, it does not seem to make a difference. I get the same error and similar desktop file results. – Argentinaargentine

request 3.0 will add automatic support for this once node v0.10 comes out – Address

Answer 1

I couldn't get request to work either, so ended up using http instead.

var http = require("http"),
    zlib = require("zlib");

function getGzipped(url, callback) {
    // buffer to store the streamed decompression
    var buffer = [];

    http.get(url, function(res) {
        // pipe the response into the gunzip to decompress
        var gunzip = zlib.createGunzip();            
        res.pipe(gunzip);

        gunzip.on('data', function(data) {
            // decompression chunk ready, add it to the buffer
            buffer.push(data.toString())

        }).on("end", function() {
            // response and decompression complete, join the buffer and return
            callback(null, buffer.join("")); 

        }).on("error", function(e) {
            callback(e);
        })
    }).on('error', function(e) {
        callback(e)
    });
}

getGzipped(url, function(err, data) {
   console.log(data);
});

Bevel

finally! I've been pover setting headers to accept gzip and trying out proxy and all kind of stuff, but this did the trick to work with the stackoverflow API! One minor thing: var gunzip = gzip.createGunzip(); should be var gunzip = zlib.createGunzip(); – Plantain

I tried using all the methods for request but failed. This one works! – Caespitose

This works but there is a much more better and easy way to do this by setting few options on request module. Please read my answer below. – Comedown

Answer 2

try adding encoding: null to the options you pass to request, this will avoid converting the downloaded body to a string and keep it in a binary buffer.

Mohur

I had the same problem, this encoding option worked for me. Thank you !! – Swirl

Answer 3

Like @Iftah said, set encoding: null.

Full example (less error handling):

request = require('request');
zlib = require('zlib');

request(url, {encoding: null}, function(err, response, body){
    if(response.headers['content-encoding'] == 'gzip'){
        zlib.gunzip(body, function(err, dezipped) {
            callback(dezipped.toString());
        });
    } else {
        callback(body);
    }
});

Persecution

Answer 4

Actually request module handles the gzip response. In order to tell the request module to decode the body argument in the callback function, We have to set the 'gzip' to true in the options. Let me explain you with an example.

Example:

var opts = {
  uri: 'some uri which return gzip data',
  gzip: true
}

request(opts, function (err, res, body) {
 // now body and res.body both will contain decoded content.
})

Note: The data you get on 'reponse' event is not decoded.

This works for me. Hope it works for you guys too.

The similar problem usually we ran into while working with request module is with JSON parsing. Let me explain it. If u want request module to automatically parse the body and provide you JSON content in the body argument. Then you have to set 'json' to true in the options.

var opts = {
  uri:'some uri that provides json data', 
  json: true
} 
request(opts, function (err, res, body) {
// body and res.body will contain json content
})

Reference: https://www.npmjs.com/package/request#requestoptions-callback

Comedown

Thank you for this!! it works and I didn't know request-promise had a gzip flag. – Moran

One caveat to setting the "gzip": true option... the server response must include a "Content-Encoding": "gzip" for the request module to actually decompress the response. I have been dealing with a server which doesn't set the "Content-Encoding" header properly, and I didn't find out this was required until I read the request module's source code. Hope this comment will help others save time trying to figure out why this isn't working if you're facing a similar situation. – Incognito

Answer 5

As seen in https://gist.github.com/miguelmota/9946206:

Both request and request-promise handle it out of the box as of Dec 2017:

var request = require('request')
  request(
    { method: 'GET'
    , uri: 'http://www.google.com'
    , gzip: true
    }
  , function (error, response, body) {
      // body is the decompressed response body
      console.log('server encoded the data as: ' + (response.headers['content-encoding'] || 'identity'))
      console.log('the decoded data is: ' + body)
    }
  )

Neurosurgery

Answer 6

I have formulated a more complete answer after trying the different ways to gunzip, and solving errors to do with encoding.

Hope this helps you too:

var request = require('request');
var zlib = require('zlib');

var options = {
  url: 'http://some.endpoint.com/api/',
  headers: {
    'X-some-headers'  : 'Some headers',
    'Accept-Encoding' : 'gzip, deflate',
  },
  encoding: null
};

request.get(options, function (error, response, body) {

  if (!error && response.statusCode == 200) {
    // If response is gzip, unzip first
    var encoding = response.headers['content-encoding']
    if (encoding && encoding.indexOf('gzip') >= 0) {
      zlib.gunzip(body, function(err, dezipped) {
        var json_string = dezipped.toString('utf-8');
        var json = JSON.parse(json_string);
        // Process the json..
      });
    } else {
      // Response is not gzipped
    }
  }

});

Precede

Answer 7

Here is my two cents worth. I had the same problem and found a cool library called concat-stream:

let request = require('request');
const zlib = require('zlib');
const concat = require('concat-stream');

request(url)
  .pipe(zlib.createGunzip())
  .pipe(concat(stringBuffer => {
    console.log(stringBuffer.toString());
  }));

Greengrocery

This was the only approach that worked for me when the remote file was actually a .gz file, pre-compressed. – Tolley

Answer 8

Here's a working example (using the request module for node) that gunzips the response

function gunzipJSON(response){

    var gunzip = zlib.createGunzip();
    var json = "";

    gunzip.on('data', function(data){
        json += data.toString();
    });

    gunzip.on('end', function(){
        parseJSON(json);
    });

    response.pipe(gunzip);
}

Full code: https://gist.github.com/0xPr0xy/5002984

Uniflorous

Thanks a bunch. I was having problems with it and used your solution, works perfectly. – Ottinger

Answer 9

I'm using node-fetch. I was getting response.body, what I really wanted was await response.text().

Alveolate

In my case, I was using response.json() so was getting some error but after using response.text() it worked. Many thanks :) – Kilah

Answer 10

With got, a request alternative, you can simply do:

got(url).then(response => {
    console.log(response.body);
});

Decompression is handled automagically when needed.

Newmann

Answer 11

I used the gunzipSync convenience method in nodejs to decompress the body. This avoids working with callbacks.

import * as zlib from "zlib";

const uncompressedBody:string = zlib.gunzipSync(body).toString("utf-8");

(in typescript)

Lousy