How to specify upload directory in multer-S3 for AWS-S3 bucket?

Question

I am using express + multer-s3 to upload files to AWS S3 service.

Using the following code, I was able to upload the files to S3 Bucket but directly in the bucket.

I want them to be uploaded in a folder inside the bucket.

I was not able to find the option to do so.

Here is the code

AWS.config.loadFromPath("path-to-credentials.json");
var s3 = new AWS.S3();

var cloudStorage = multerS3({
    s3: s3,
    bucket: "sample_bucket_name",
    contentType: multerS3.AUTO_CONTENT_TYPE,
    metadata: function(request, file, ab_callback) {
        ab_callback(null, {fieldname: file.fieldname});
    },
    key: function(request, file, ab_callback) {
        var newFileName = Date.now() + "-" + file.originalname;
        ab_callback(null, newFileName);
    },
});
var upload = multer({
    storage: cloudStorage
});

router.post("/upload", upload.single('myFeildName'), function(request, response) {
    var file = request.file;
    console.log(request.file);
    response.send("aatman is awesome!");
});

Miriam

Answer 1

S3 doesn't always have folders (see http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html). It will simulate folders by adding a strings separated by / to your filename.

e.g.

key: function(request, file, ab_callback) {
    var newFileName = Date.now() + "-" + file.originalname;
    var fullPath = 'firstpart/secondpart/'+ newFileName;
    ab_callback(null, fullPath);
},

Kassie

Thank you so much!! Worked like charm... :) – Miriam

Is there a way to make the 'firstpart/secondpart' prefix dynamic? I have used req.params.destination but no luck – Agretha

@HerveTribouilloy absolutely. The firstpart/secondpart are just strings. You can generate them however you want. If you are not seeing what you expect first make sure you actually have a value in your params. – Kassie

yes, my issue is I can't seem to send a param to the request parameter that multer/multerS3 uses. I have also read that multer does not handle params well, so I was wondering if you have more experience on this, thanks for your feedback – Agretha

@HerveTribouilloy the request passed to multer is not the same as what is processed by the router. You have the full path though and can parse it to get the desired param. – Kassie

thanks, the problem was due to the POST request data were not being parsed. I have not found a way to parse them but instead, I found a workaround that consist in using the query parameters (the file to upload in in POST data and the other parameters are in the query). thanks for your support – Agretha

Answer 2

My solution to dynamic destination path. Hope this helps somebody!

const fileUpload = function upload(destinationPath) {
  return multer({
    fileFilter: (req, file, cb) => {
      const isValid = !!MIME_TYPE_MAP[file.mimetype];
      let error = isValid ? null : new Error("Invalid mime type!");
      cb(error, isValid);
    },
    storage: multerS3({
      limits: 500000,
      acl: "public-read",
      s3,
      bucket: YOUR_BUCKET_NAME,
      contentType: multerS3.AUTO_CONTENT_TYPE,
      metadata: function (req, file, cb) {
        cb(null, { fieldName: file.fieldname });
      },
      key: function (req, file, cb) {
        cb(null, destinationPath + "/" + file.originalname);
      },
    }),
  });
};



module.exports = fileUpload;

How to call:

router.patch(
  "/updateProfilePicture/:userID",
  fileUpload("user").single("profileimage"),
  usersControllers.updateProfilePicture
);

"profile image" is the key for the file passed in the body.
"user" is path to the destination folder. You can pass any path, consisting of folder and sub folders. So this puts my file in a folder called "user" inside my bucket.

Bowdlerize

FYI the limits property should go in the multer{} object with the fileFilter property, not the multers3{} object – Galaxy

Answer 3

Multer s3 uses a string path to add folders. Also, you can utilize req and file objects to add dynamic paths. But in the req object, you can only utilize headers, params, and query properties, not the body property as the data is not processed yet. Also, don't try to use 2 multer's in the request it will not work. Here, use the middleware and functions to preprocess using the req object and use the req object to add the customized path as the request object is a type of reference object, thus the changes in 1 middleware will reflect on its proceeding middleware.

It is the only possible solution according to me.

[EDIT 1]
Example of the above approach:
Required Path:

<s3-bucket-host-url>/user/:id/<file-name>

Middleware Code:

const preprocessMiddleware = (req, res, next) => {
let path = 'users/'; //Add static path
path = path + req.params.id;//Add id
req.processedpath = path;
next();
}

Inside Multer s3 storage function:

const storage = multerS3({
  s3: s3,
  acl: <S3ACL>,
  bucket: <S3BUCKETNAME>,
  key: (req, file, cb) => {
    let path = req.processedpath;
    const trimSplash = (str) => str.replace(/^\s*\/*\s*|\s*\/*\s*$/gm, ''); //The function is to trim all the leading and trailing slashes
    const newFileName = Date.now() + "-" + file.originalname;
    const fullPath = trimSplash (path + '/' + newFileName);
    cb(null, fullPath); //use Date.now() for unique file keys
  }
});

Router should look like this:

router.post('/uploads3', preprocessMiddleware, uploader.single('document'), someothermiddlewares);

Eberta

Thank you. The people will hardly understand. Better add an example where the key function uses the req argument. – Canicular

Answer 4

There is another nice way of doing it! After you write your bucket name, add the folder name there itself with a slash infront.

const multerS3Config = multerS3({
    s3: s3,
    bucket: process.env.AWS_BUCKET_NAME + '/<your-folder-name>', //put a slash infront
    metadata: function (req, file, cb) {
        cb(null, { fieldName: file.fieldname });
    },
    key: function (req, file, cb) {
    cb(null, Date.now() + file.originalname)
    }  
});

Biradial