How to buffer an HTTP response using the request module?

Question

I would like to stream the contents of an HTTP response to a variable. My goal is to get an image via request(), and store it in in MongoDB - but the image is always corrupted.

This is my code:

request('http://google.com/doodle.png', function (error, response, body) {
    image = new Buffer(body, 'binary');
    db.images.insert({ filename: 'google.png', imgData: image}, function (err) {
        // handle errors etc.
    });
})

What is the best way to use Buffer/streams in this case?

Barone

Answer 1

The request module buffers the response for you. In the callback, body is a string (or Buffer).

You only get a stream back from request if you don't provide a callback; request() returns a Stream.

See the docs for more detail and examples.

---

request assumes that the response is text, so it tries to convert the response body into a sring (regardless of the MIME type). This will corrupt binary data. If you want to get the raw bytes, specify a null encoding.

request({url:'http://google.com/doodle.png', encoding:null}, function (error, response, body) {
    db.images.insert({ filename: 'google.png', imgData: body}, function (err) {

        // handle errors etc.

    }); 
});

Glim

Thanks for the calification. I have updated my question. Please see if you can expand your answer. – Barone

Wow. Thank You, it worked. That's been driving me nuts for so long. – Barone

thx this works, but I see that when passing 'encoding: null', in the callback, body is Buffer and not string. to me this is the primary reason why this works. – Vetiver

Answer 2

var options = {
    headers: {
        'Content-Length': contentLength,
        'Content-Type': 'application/octet-stream'
    },
    url: 'http://localhost:3000/lottery/lt',
    body: formData,
    encoding: null, // make response body to Buffer.
    method: 'POST'
};

set encoding to null, return Buffer.

Marbles

In my case the encoding: null resolved my issue. I was trying to send a pdf buffer which was the response from a serveless function that created the pdf buffer – Subir

Answer 3

Have you tried piping this?:

request.get('http://google.com/doodle.png').pipe(request.put('{your mongo path}'))

(Though not familiar enough with Mongo to know if it supports direct inserts of binary data like this, I know CouchDB and Riak do.)

Transpontine

Thanks but I don't want to use pipe. It's possible to stream directly into mongodb but that will mean I will have to use Gridfs which I want to avoid. – Barone

This is HTTP Rest, but you can pipe to a local file too: github.com/mikeal/request#streaming – Transpontine

Answer 4

Nowadays, you can easily retreive a file in binary with Node 8, RequestJS and async await. I used the following:

const buffer = await request.get(pdf.url, { encoding: null }); 

The response was a Buffer containing the bytes of the pdf. Much cleaner than big option objects and old skool callbacks.

Wingover