How to store a binary object in redis using node?

Question

I am trying to save a binary object in redis and then serve it back as an image.

Here is the code I am using to save the data:

var buff=new Buffer(data.data,'base64');
client.set(key,new Buffer(data.data,'base64'));

Here is the code to dump the data out:

client.get(key,function(err,reply){
        var data = reply;
        response.writeHead(200, {"Content-Type": "image/png"});
        response.end(data,'binary');

});

The first few byte of the data seem to be corrupted. The magic number is incorrect.

Did some experimenting:

when I do the following:

var buff=new Buffer(data.data,'base64');
console.log(buff.toString('binary'));

I get this:

0000000: c289 504e 470d 0a1a 0a00 0000 0d49 4844

when I do this

 var buff=new Buffer(data.data,'base64');
 console.log(buff);

I get the following:

Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 00

I am not sure where the c2 is coming from

Windy

haven't worked with images this way, but isn't supposed to "decode" what was encoded in base64? – Minute

What do you get if you console.log(reply)? Can you also elaborate on what you are trying accomplish by putting the images in Redis? It is not the best way to store large amounts of binary data. – Kelm

data comes in as base64 trying to save the binary file – Windy

Follow up to this question: I'm trying to do basically the same thing, and when I read it out with get, the string length is different even when I'm writing the same thing. – Clench

Answer 1

The problem is that the Redis client for Node converts responses to JavaScript strings by default.

I solved this by setting the return_buffers option to true when creating the client.

var client = redis.createClient(7000, '127.0.0.1', {'return_buffers': true});

See here for more details.

Camerlengo

Answer 2

I ran in to a similar issue and in my case the corruption was with how I was getting data from Redis into Nodejs. In my case the value from Nodejs was stored correctly but when it came into node it was being converted to a ASCII string. I was able to force it to a buffer with the following line.

let file = await r.get(redis.commandOptions({ returnBuffers: true }), `zip/${req.params.taskId}`)

Sixpence

Answer 3

The problem with return_buffers is when you are not using pure buffer data's then you'll have to do something to convert other buffers to strings. While detect_buffers could be an option it is too unreliable.

If you don't mind the extra computing cycles. You could also try:

// convert your buffer to hex string
client.set(key, mybuffer.toString('hex'));
// create a new buffer from hex string
client.get(key, function (err, val) {
   var bin = new Buffer(val, 'hex');
});

Railway

In most cases, base64 is a better option since it will produce a smaller message than hex (at a cost of more expensive encoding). – Skim

Answer 4

I was unable to figure out how to get the binary string to store.

Here is my workaround:

Where data is the data in base64 string

client.set(count,data);

to serve the data:

 client.get(last,function(err,reply){
   var data = reply;
   response.writeHead(200, {"Content-Type": "image/png"});
   var buff=new Buffer(data,'base64');
   response.end(buff);
});

This isn't ideal since you have to do the conversion every time, but it seem to work.

Windy

Please note, that new Buffer() is deprecated. Use: Buffer.from('some-data', 'base64'); – Gordon

Answer 5

This post is old but just an updated answer. As another answer stated all Node.js Redis libraries return data as utf8 strings because thats what people generally need. Node.js Redis libraries like ioRedis also provide alternatives to fetch data as a buffer. For example redis.getBuffer instead of redis.get.

Ecclesiastical

Answer 6

What worked for me is to use data.toString('binary') if it is a Buffer. Also make sure not to reinterpret it as utf-8, but also as binary.

Philae

Answer 7

I found this article on the topic which explains the implications of doing this, which I highly recommend reading before going down this path (regardless of programming language).

http://qnimate.com/storing-binary-data-in-redis/

In summary, since Redis is an in-memory key/value store, you would be wise to not store potentially large values such as images inside Redis as you will quickly use up available memory and degrade the performance of your Redis instance. It is better to store the location of the file in Redis rather than the file itself.

Swabber