How to store files with meta data in LoopBack?

Question

What I want to do: Have an html form, with a file input inside. When a file is chosen, the file input should upload the file, and get a file id, so when the form is submitted, the file id is posted with the form and written in the database.

Shorter version: I want to store meta data (id for example) with my files.

Sounds simple, yet I struggle to do that in LoopBack.

There has been a couple conversations ( 1, 2 ) about this topic, and neither seemed to lead to a solution, so I thought this might be a good place to find one once and for all.

The simplest solution would be to use model relations, but LoopBack doesn't support relations with the file storage service. Bump. So we have to go with a persistedmodel named File for example, and override default create, delete so it saves and deletes from the file store model I have - named Storage.

My setup so far:

• I have a model /api/Storage which is connected to a loopback storage service and is saving file successfully to the local filesystem.
• I have a PersistedModel connected to Mongo with file meta data: name,size, url and objectId
• I have a remote hook set up beforecreate so the file can be saved first and then it's url can be injected into File.create()

I'm there, and according to this LoopBack page, I have the ctx which should have the file inside:

File.beforeRemote('create', function(ctx, affectedModelInstance, next) {})`

What's ctx?

ctx.req: Express Request object.
ctx.result: Express Response object.

Ok, so now I'm at the Express page, pretty lost, and it sais something about a 'body-parsing middleware' which I have no idea what it might be.

I feel like I'm close to the solution, any help would be appreciated. Is this approach right?

Vitrain

I can get data for File.beforeRemote('upload', function(ctx, modelInstance, next){ console.log(ctx.req); next(); }); , however I can't see any file related information in the ctx object, and the modelInstance is undefined too... Worth noting that my File here is the model with the storage service datasource. – Clearsighted

Thanks RYFN for taking a look into this. For the sake of consistency, I'll stick with my naming 'File' for the file meta data and storageId, and 'Storage' for the file model bound to storage service. – Vitrain

I can easily do a remote hook to Storage.upload, and get file meta data like name, size, etc., and call File.create() from the hook, but this is not the best solution. File being a persistentModel can be set to be related to User.profileimage for example, and if a user posts a form with the image inside, it would be handled well by Loopback. So I'm still looking for a solution with a hook to File and not Storage models. – Vitrain

how do you get the file meta data out of the .upload hook? would you be able to show an example? – Clearsighted

Storage.afterRemote('upload',function(ctx, modelInstance, next){ console.log('create file',modelInstance.result.files.file); next(); }); – Vitrain

Answer 1

Here's the full solution for storing meta data with files in loopback.

You need a container model

common/models/container.json

{
  "name": "container",
  "base": "Model",
  "idInjection": true,
  "options": {
    "validateUpsert": true
  },
  "properties": {},
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}

Create the data source for your container in server/datasources.json. For example:

...
"storage": {
    "name": "storage",
    "connector": "loopback-component-storage",
    "provider": "filesystem", 
    "root": "/var/www/storage",
    "maxFileSize": "52428800"
}
...

You'll need to set the data source of this model in server/model-config.json to the loopback-component-storage you have:

...
"container": {
    "dataSource": "storage",
    "public": true
}
...

You'll also need a file model to store the meta data and handle container calls:

common/models/files.json

{
  "name": "files",
  "base": "PersistedModel",
  "idInjection": true,
  "options": {
    "validateUpsert": true
  },
  "properties": {
    "name": {
      "type": "string"
    },
    "type": {
      "type": "string"
    },
    "url": {
      "type": "string",
      "required": true
    }
  },
  "validations": [],
  "relations": {},
  "acls": [],
  "methods": []
}

And now connect files with container:

common/models/files.js

var CONTAINERS_URL = '/api/containers/';
module.exports = function(Files) {

    Files.upload = function (ctx,options,cb) {
        if(!options) options = {};
        ctx.req.params.container = 'common';
        Files.app.models.container.upload(ctx.req,ctx.result,options,function (err,fileObj) {
            if(err) {
                cb(err);
            } else {
                var fileInfo = fileObj.files.file[0];
                Files.create({
                    name: fileInfo.name,
                    type: fileInfo.type,
                    container: fileInfo.container,
                    url: CONTAINERS_URL+fileInfo.container+'/download/'+fileInfo.name
                },function (err,obj) {
                    if (err !== null) {
                        cb(err);
                    } else {
                        cb(null, obj);
                    }
                });
            }
        });
    };

    Files.remoteMethod(
        'upload',
        {
            description: 'Uploads a file',
            accepts: [
                { arg: 'ctx', type: 'object', http: { source:'context' } },
                { arg: 'options', type: 'object', http:{ source: 'query'} }
            ],
            returns: {
                arg: 'fileObject', type: 'object', root: true
            },
            http: {verb: 'post'}
        }
    );

};

For expose the files api add to the model-config.json file the files model, remember select your correct datasources:

...
"files": {
    "dataSource": "db",
    "public": true
}
...

Done! You can now call POST /api/files/upload with a file binary data in file form field. You'll get back id, name, type, and url in return.

Vitrain

File.app is not defined :( – Canoness

make sure you take File as the attribute for your hook module.exports = function(File) {...} – Vitrain

Hey I get a response saying [Error: Request aborted]. I have done exactly what u have done. Any leads would be awsm. thanks. – Blisse

I wonder if CouchDB would be a solution here - you can save attachments with documents in couch just nicely – Arteriovenous

@Vaibhav I get the same error as you. Did you ever figure it out? – Prevision

@Vaibhav I figured it out. The documentation (docs.strongloop.com/display/public/LB/Storage+component) for upload says (Upload one or more files into the specified container. The request body must use multipart/form-data which the file input type for HTML uses). Once I did this everything went to work. – Prevision

I'm getting 404, when uplaoding image on this url POST localhost:3004/api/v1/File/upload are we needed expose file model in datasource.json too. – Thi

Is it the optimal choice, given the fact that options are sent as query string? – Clere

Does anyone have code for the Android/iOS SDK side to upload a File with metadata?? – Uniformitarian

What if file upload to container is successful but File.create() fails? we will have an orphan file in the container with no entry in our database (what is the main purpose of this). Wouln't be proper to delete the file in this case? – Milfordmilhaud

File.app.models.container.upload does not process file for me and I get "request aborted" error due to timeout. as I described in #39889971 I couldn't fix the problem. could anyone help me? – Dimitrovo

Can I rename file before it gets uploaded? – Khadijahkhai

@NileshG, sure in File.create({ name: fileInfo.name, type: fileInfo.type, container: fileInfo.container, url: CONTAINERS_URL+fileInfo.container+'/download/'+fileInfo.name },..... you can set name to whatever. – Vitrain

I've the exact same problem that the upload function hangs and returns an empty response. As I'm using loopback 3, everything mentioned here seems not to work! Any ideas for loopback 3? – Nuncupative

@Nuncupative yes, the solution is for Loopback 2 - please note that the ticket was added and solved 2 years ago (somewhere in the begninning of 2015), when Loopback3 wasn't out. If you find a solution for Loopback 3, feel free to link it here, or add as an anwser. Thanks, M – Vitrain

the above config give me .('Cannot override built-in "{{file}}" type.')); – Cubiform

I want to send one more param with file like mobile param, I don't know how to get that request param, please any one. I tried ctx.req.mobile but didn't worked. – Abebi

Answer 2

I had the same problem. I solved it by creating my own models to store meta data and my own upload methods.

1. I created a model File which will store info like name,type,url,userId ( same as yours)

2. I created my own upload remote method because I was unable to do it with the hooks. Container model is the model which is created by loopback-component-storage.

3. var fileInfo = fileObj.files.myFile[0]; Here myFile is the fieldname for file upload, so you will have to change it accordingly. If you don't specify any field, then it will come as fileObj.file.null[0]. This code lacks proper error checking, do it before deploying it in production.

    File.uploadFile = function (ctx,options,cb) {
     File.app.models.container.upload(ctx.req,ctx.result,options,function (err,fileObj) {
       if(err) cb(err);
       else{
               // Here myFile is the field name associated with upload. You should change it to something else if you
               var fileInfo = fileObj.files.myFile[0];
               File.create({
                 name: fileInfo.name,
                 type: fileInfo.type,
                 container: fileInfo.container,
                 userId: ctx.req.accessToken.userId,
                 url: CONTAINERS_URL+fileInfo.container+'/download/'+fileInfo.name // This is a hack for creating links
               },function (err,obj) {
                 if(err){
                   console.log('Error in uploading' + err);
                   cb(err);
                 }
                 else{
                   cb(null,obj);
                 }
               });
             }
           });
   };
   
   File.remoteMethod(
     'uploadFile',
     {
       description: 'Uploads a file',
       accepts: [
       { arg: 'ctx', type: 'object', http: { source:'context' } },
       { arg: 'options', type 'object', http:{ source: 'query'} }
       ],
       returns: {
         arg: 'fileObject', type: 'object', root: true
       },
       http: {verb: 'post'}
     }
   
   );
   

Grave

Awsome, your answer guided me to the solutions I was looking for. I'll accept yours, and will upload the full solution with model defs for the record. – Vitrain

great! for any one like me who got ( can't find upload of undefined ) make sure your container model is named container small case! or change it! – Frugivorous

Answer 3

For those who are looking for an answer to the question "how to check file format before uploading a file".

Actual in this case we can use optional param allowedContentTypes.

In directory boot use example code:

module.exports = function(server) {
    server.dataSources.filestorage.connector.allowedContentTypes = ["image/jpg", "image/jpeg", "image/png"];
}

I hope it will help someone.

Teutonism

Answer 4

Depending on your scenario, it may be worth looking at utilising signatures or similar allowing direct uploads to Amazon S3, TransloadIT (for image processing) or similar services.

Our first decision with this concept was that, as we are using GraphQL, we wanted to avoid multipart form uploads via GraphQL which in turn would need to transfer to our Loopback services behind it. Additionally we wanted to keep these servers efficient without potentially tying up resources with (large) uploads and associated file validation and processing.

Your workflow might look something like this:

1. Create database record
2. Return record ID and file upload signature data (includes S3 bucket or TransloadIT endpoint, plus any auth tokens)
3. Client uploads to endpoint

For cases where doing things like banner or avatar uploads, step 1 already exists so we skip that step.

Additionally you can then add SNS or SQS notifications to your S3 buckets to confirm in your database that the relevant object now has a file attached - effectively Step 4.

This is a multi-step process but can work well removing the need to handle file uploads within your core API. So far this is working well from our initial implementation (early days in this project) for things like user avatars and attaching PDFs to a record.

Example references:

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingHTTPPOST.html

https://transloadit.com/docs/#authentication

Solemnize

Answer 5

For anyone else having that problem with loopback 3 and Postman that on POST, the connection hangs (or returns ERR_EMPTY_RESPONSE) (seen in some comments here)... The problem in this scenario is, that Postman uses as Content-Type "application/x-www-form-urlencoded"!

Please remove that header and add "Accept" = "multipart/form-data". I've already filed a bug at loopback for this behavior

Nuncupative

Yes of course, you can find it here: github.com/strongloop/loopback-component-storage/issues/196 – Nuncupative

"The solution to the problem is to explicitly set Content-Type to undefined so that your browser or whatever client you're using can set it and add that boundary value in there for you. Disappointing but true." - #39280938 – Ibis

Answer 6

Just pass the data as "params" object and at server you can get it as ctx.req.query

For example

At client side

Upload.upload(
{
    url: '/api/containers/container_name/upload',
    file: file,
    //Additional data with file
    params:{
     orderId: 1, 
     customerId: 1,
     otherImageInfo:[]
    }
});

At Server side

Suppose your storage model name is container

Container.beforeRemote('upload', function(ctx,  modelInstance, next) {
    //OUPTUTS: {orderId:1, customerId:1, otherImageInfo:[]}
    console.log(ctx.req.query); 
    next();
})

Kirven

Thanks Robins for taking the time to reply. Good point, but the solution you proposed doesn't address the main problem: How will you store and return this data with the file url from the same api (in your case /api/containers/container_name/file). Harshil's solution was closer to what I was looking for. Thanks for your contribution. – Vitrain

@MihalyKR i think that this approach could be work. At the time that you upload a file in the container model you recieve in the response the obj providerResponse: {... ,"location": ".."} ,I was thinking in use this location inside a hook beforeCreate and setting this in the url of the File. So in just one method you define the binary in your storage and the metadata for your persistedmodel. – Deandre

Answer 7

For AngularJS SDK users... In case you would like to use generated methods like Container.upload(), you might want to add a line to configure the method in lb-services.js to set Content-Type headers to undefined. This would allow client to set Content-Type headers and add boundary value automatically. Would look something like this:

 "upload": {
    url: urlBase + "/containers/:container/upload",
    method: "POST",
    headers: {"Content-Type": undefined}
 }

Ralina