Multer-S3 with sharp upload and resize

Question

I've been able to upload to s3 an image file using multer-s3 library but now I'm trying to add the sharp library to resize and rotate but can't figure out where or how it needs to go. This is my current code that works for uploading:

var options = {
'endpoint' : 'https://xxx',
'accessKeyId' : '123',
'secretAccessKey' : '123abc',
'region' : 'xxx'
};

 const s3 = new aws.S3(options);

const upload = multer({
storage: multerS3({
    s3: s3,
    bucket: 'wave',
    acl: 'public-read',
    metadata: function (req, file, cb) {
      cb(null, Object.assign({}, req.body));
    },
    key: function (request, file, cb) {
        console.log('inside multerS3',file);
        cb(null, file.originalname);
    }
})
}).array('file', 1);

app.post('/upload/', (req,res) => {
   upload(req,res, async function (error) {
     if(error){
        res.send({ status : error });
     }else{
        console.log(req.files[0]);
        res.send({ status : 'true' });
     }
    });
 });

And do something like this with the file:

 sharp(input)
 .resize({ width: 100 })
 .toBuffer()
 .then(data => {
   // 100 pixels wide, auto-scaled height
 });

Any help would be greatly appreciated :)

Sapota

did you find a solution ? if so answer your own question. thanks – Wont

I have your EXACT same problem. PLEASE if you've found a solution post it down here! – Secondly

Answer 1

Ok so I never found an actual solution to this but I did get a recommendation which is the route I went with. Because of performance considering multiple uploads by multiple users to do a resize, rotate and whatever else to the image at time of upload is not the best solution because that would be heaving on the server. The way we went with was have a cdn server and on that server have a script that detects when a new file is uploaded with goes ahead and does resizing and rotating.

I know it's not an actual fix to the original problem but it was the best solution I got considering performance and optimization.

Sapota

What CDN service did you use? – Bachelor

Answer 2

I tried lots of different multer-s3 packages with Sharp support but none of them worked for me so I decided to go with multer, aws-sdk and sharp.

Here's a working Express.js example.

const path = require('path')
const sharp = require('sharp')
const AWS = require('aws-sdk')
const multer = require('multer')
const express = require('express')
require('dotenv').config({ path: path.join(__dirname, '.env') })

const { AWS_BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT } = process.env

const app = express()
const port = 9000

app.use(express.json())
app.listen(port, () => {
  console.log('Example app listening on port http://localhost:' + port)
})

app.get('/', (req, res) => {
  res.send(`
<form action='/upload' method='post' enctype='multipart/form-data'>
  <input type='file' name='file' multiple />
  <input type='submit' value='upload' />
</form>
  `)
})

const sharpify = async originalFile => {
  try {
    const image = sharp(originalFile.buffer)
    const meta = await image.metadata()
    const { format } = meta
    const config = {
      jpeg: { quality: 80 },
      webp: { quality: 80 },
      png: { quality: 80 }
    }
    const newFile = await image[format](config[format])
      .resize({ width: 1000, withoutEnlargement: true })
    return newFile
  } catch (err) {
    throw new Error(err)
  }
}

const uploadToAWS = props => {
  return new Promise((resolve, reject) => {
    const s3 = new AWS.S3({
      accessKeyId: AWS_ACCESS_KEY_ID,
      secretAccessKey: AWS_SECRET_ACCESS_KEY,
      endpoint: new AWS.Endpoint(AWS_ENDPOINT)
    })
    s3.upload(props, (err, data) => {
      if (err) reject(err)
      resolve(data)
    })
  })
}

app.post('/upload', multer().fields([{ name: 'file' }]), async (req, res) => {
  try {
    const files = req.files.file
    for (const key in files) {
      const originalFile = files[key]

      const newFile = await sharpify(originalFile)

      await uploadToAWS({
        Body: newFile,
        ACL: 'public-read',
        Bucket: AWS_BUCKET,
        ContentType: originalFile.mimetype,
        Key: `directory/${originalFile.originalname}`
      })
    }

    res.json({ success: true })
  } catch (err) {
    res.json({ success: false, error: err.message })
  }
})

Liborio

Answer 3

Edit (12/24/2020): I can no longer recommend this package and I'm no longer using it in my projects. This package lacks many of the other features multerS3 has, including AUTO_CONTENT_TYPE which I use often to automatically set the content-type in S3. I highly recommend just sticking to multerS3 or not even using multerS3 at all and handling the multipart form data yourself with something like multiparty.

Original post: I found this package on npm called multer-sharp-s3 Link: https://www.npmjs.com/package/multer-sharp-s3.

I'm using this implementation:

var upload = multer({
    storage: multerS3({
        s3: s3,
        ACL: 'public-read',
        Bucket: s3Bucket,
        Key: (req, file, cb) => {
            cb(null, file.originalname);
        },
        resize: {
            width: 100,
            height: 100
        }
    }),
    fileFilter: fileFilter
});

Hope this helps!

Viscountcy

This package cannot be installed on Apple M1 chips because it's using an old version of Sharp. And looks like it's not maintained anymore. – Liborio

Answer 4

You can add contentType method to multerS3 options and pipe sharp to file stream. Also you can combine it with default autoContentType function from multerS3 lib (link to autoContentType).

Here is my example:

contentType(
  req: Express.Request,
  file: Express.Multer.File,
  callback: (
    error: any,
    mime?: string, 
    stream?: NodeJS.ReadableStream,
  ) => void,
) {
  const mime = 'application/octet-stream';
  const outStream = sharp().webp({ quality: 80 });

  file.stream.pipe(outStream);

  callback(null, mime, outStream);
}

Harding

Answer 5

Artur mentioned the contentType method which is what I recommend. However, his uses TypeScript so here is my full Multer-S3 configuration (converts all jpeg, png, and jpg to webp) in javascript for those that want a little copy paste action:

const imageStorage = multerS3({
    s3: s3,
    acl: 'private',
    bucket: process.env.AWS_BUCKET_NAME, 
    contentType: function(req, file, callback) {
        const mime = 'application/octet-stream';
        const outStream = sharp().webp({ quality: 60 });

        file.stream.pipe(outStream);

        callback(null, mime, outStream);
    },
    key: function (req, file, cb) {
        //Uploads file to uploadDirectory (specified by route middleware) and fileKey (specified by route middleware with Date.now())
        const target = "images/" + Date.now() + ".webp"; 
        cb(null, target); 
    }
});

I will mention though that this will increase your upload speed by a few seconds (or more depending on file size), if that's an issue for you, as others have recommended, I would use CDNs.

Superstitious