showing an image from amazon s3 with nodejs, expressjs and knox

Question

I think this should be a straight forward thing, but I can't fing a solution :s

I'm trying to figure out the best way to show images stored on amazon S3 on a website.

Currently I'm trying to get this to work (unsuccessful)

//app.js
app.get('/test', function (req, res) {
    var file = fs.createWriteStream('slash-s3.jpg');
    client.getFile('guitarists/cAtiPkr.jpg', function(err, res) {
        res.on('data', function(data) { file.write(data); });
        res.on('end', function(chunk) { file.end(); });
    });
});

//index.html
<img src="/test" />

Isn't it maybe possible to show the images directly from amazon ? I mean, the solution that lightens the load on my server would be the best.

Maes

Answer 1

This is a typical use case for streams. What you want to do is: request a file from Amazon S3 and redirect the answer of this request (ie. the image) directly to the client, without storing a temporary file. This can be done by using the .pipe() function of a stream.

I assume the library you use to query Amazon S3 returns a stream, as you already use .on('data') and .on('end'), which are standard events for a stream object.

Here is how you can do it:

app.get('/test', function (req, res) {
    client.getFile('guitarists/cAtiPkr.jpg', function(err, imageStream) {
        imageStream.pipe(res);
    });
});

By using pipe, we redirect the output of the request to S3 directly to the client. When the request to S3 closes, this will automatically end the res of express.

For more information about streams, refer to substack's excellent Stream Handbook.

PS: Be careful, in your code snippet you have two variables named res: the inner variable will mask the outer variable which could lead to hard to find bugs.

Homocyclic

Your server makes the actual request, but it won't store the result to a file and instead will directly write the output to the client's socket. This is very efficient and allows for neat optimizations, refer to the stream handbook for an overview of the core streaming mechanisms. – Homocyclic

How can you apply this method when streaming multiple images? For instance, streaming many images onto the user's new feed like Facebook. – Antistrophe

Answer 2

if you set the access controls correctly (on a per-key basis. not on the whole Bucket.) then you can just use <img src="http://aws.amazon.com/myBucket/myKey/moreKey.jpg"> (or another appropriate domain if you're using something other than us-east-1) wherever you want to display the image in your html. remember to set the MIME type if you want browsers to display the image instead of downloading the image as attachment when someone opens it.

aws-sdk docs
s3: PUT Object docs

var AWS = require('aws-sdk');
AWS.config.update({
  accessKeyId: "something",
  secretAccessKey: "something else",
  region:'us-east-1',
  sslEnabled: true,
});
var fileStream = fs.createReadStream(filename);
s3.putObject({
  Bucket:'myBucket',
  Key: 'myKey/moreKey.jpg',
  Body: fileStream,
  ContentType:'image/jpeg',
  ACL: 'public-read',
}, function(err, data){
  if(err){ return callback(err) };
  console.log('Uploaded '+key);
  callback(null);
});

Biotype