How to perform an HTTP file upload using express on Cloud Functions for Firebase (multer, busboy)

Question

I am trying to upload a file to Cloud Functions, using Express to handle requests there, but i am not succeeding. I created a version that works locally:

serverside js

const express = require('express');
const cors = require('cors');
const fileUpload = require('express-fileupload');

const app = express();
app.use(fileUpload());
app.use(cors());

app.post('/upload', (req, res) => {
    res.send('files: ' + Object.keys(req.files).join(', '));
});

clientside js

const formData = new FormData();
Array.from(this.$refs.fileSelect.files).forEach((file, index) => {
    formData.append('sample' + index, file, 'sample');
});

axios.post(
    url,
    formData, 
    {
        headers: { 'Content-Type': 'multipart/form-data' },
    }
);

This exact same code seems to break when deployed to Cloud Functions, where req.files is undefined. Does anyone have any idea what is happening here?

EDIT I also had a go at using multer, which worked fine locally, but once uploaded to Cloud Functions, this got me an empty array (same clientside code):

const app = express();
const upload = multer();
app.use(cors());

app.post('/upload', upload.any(), (req, res) => {
    res.send(JSON.stringify(req.files));
});

Coquet

I don't know about express-fileupload, but I've used the multer module to receive file uploads successfully. – Settles

Do you have a working example of that? I did try multer, and express-form-data but for some reason didn't have success with any of them. – Coquet

I tried setting something up locally with multer, and it worked immediately (see edit in post). But again, when deploying to Firebase i get no result. I don't understand what i am doing wrong. – Coquet

@Coquet I ran into the same problem as you, but I have successfully deployed Cloud Functions in the past that accept multipart input, and the exact same code does not read any input when deployed now! (into a new function) I have been pulling my hair out the last day trying to find the source of this issue, switching from using formidable, multiparty, busboy, multer, etc. all of them returning empty parses. I now think that there was a silent update in Google Cloud Functions that broke it all... Which is why my older versions still work but the new apis don't... just guessing – Preeminence

@Coquet PS: I will have to continue trying to solve this riddle now, and will keep you posted if I find any clues. Please let me know aswell if you solve it. – Preeminence

OK, I'm running into the same problem with my multer code that worked OK in the past. Works fine locally with firebase serve, however. Will investigate with the Cloud Functions team. @Preeminence @Coquet – Settles

@DougStevenson So it indeed seems that this was an update on their side that broke our codes. Thank you for trying to get in contact with them Doug. I never managed to ask a question to a Google Cloud employee directly – Preeminence

I contacted with firebase-support and I got an answer: "It seems that Cloud Functions might be having an issue with multipart/form-data. I'll be escalating your case to our engineers and I'll let you know once I get an update." – Oldline

i have opened a issue here => github.com/firebase/firebase-functions/issues/141 – Ergograph

can guys have any example for using multer for firebase storage in firebase cloud function – Surrejoinder

@DougStevenson i see your comment in another question that multer doesn't work with cloud functions, i need to know why it doesn't work? – Surrejoinder

Answer 1

There was indeed a breaking change in the Cloud Functions setup that triggered this issue. It has to do with the way the middleware works that gets applied to all Express apps (including the default app) used to serve HTTPS functions. Basically, Cloud Functions will parse the body of the request and decide what to do with it, leaving the raw contents of the body in a Buffer in req.rawBody. You can use this to directly parse your multipart content, but you can't do it with middleware (like multer).

Instead, you can use a module called busboy to deal with the raw body content directly. It can accept the rawBody buffer and will call you back with the files it found. Here is some sample code that will iterate all the uploaded content, save them as files, then delete them. You'll obviously want to do something more useful.

const path = require('path');
const os = require('os');
const fs = require('fs');
const Busboy = require('busboy');

exports.upload = functions.https.onRequest((req, res) => {
    if (req.method === 'POST') {
        const busboy = new Busboy({ headers: req.headers });
        // This object will accumulate all the uploaded files, keyed by their name
        const uploads = {}

        // This callback will be invoked for each file uploaded
        busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
            console.log(`File [${fieldname}] filename: ${filename}, encoding: ${encoding}, mimetype: ${mimetype}`);
            // Note that os.tmpdir() is an in-memory file system, so should only 
            // be used for files small enough to fit in memory.
            const filepath = path.join(os.tmpdir(), fieldname);
            uploads[fieldname] = { file: filepath }
            console.log(`Saving '${fieldname}' to ${filepath}`);
            file.pipe(fs.createWriteStream(filepath));
        });

        // This callback will be invoked after all uploaded files are saved.
        busboy.on('finish', () => {
            for (const name in uploads) {
                const upload = uploads[name];
                const file = upload.file;
                res.write(`${file}\n`);
                fs.unlinkSync(file);
            }
            res.end();
        });

        // The raw bytes of the upload will be in req.rawBody.  Send it to busboy, and get
        // a callback when it's finished.
        busboy.end(req.rawBody);
    } else {
        // Client error - only support POST
        res.status(405).end();
    }
})

Bear in mind that files saved to temp space occupy memory, so their sizes should be limited to a total of 10MB. For larger files, you should upload those to Cloud Storage and process them with a storage trigger.

Also bear in mind that the default selection of middleware added by Cloud Functions is not currently added to the local emulator via firebase serve. So this sample will not work (rawBody won't be available) in that case.

The team is working on updating the documentation to be more clear about what all happens during HTTPS requests that's different than a standard Express app.

Settles

I've tried your solution, but I'm trying to use busboy's 'field' callback to collect body content. But it seems the field event is never fired, and then finish is never fired either, and the request ends up timing out. – Norahnorbert

I have just tried this solution (by Google's recommendation), but it does not work for me. It does not work for me. Here is the result: The client (Chrome, fetch) logs the call to the local Firebase server as "multipart/form-data", with the file I want to upload and a field. The Firebase server does not show any call to "busboy.on('file'...)". There is a call to "busboy.on('finish', ...)", but the "uploads" array is empty. – Roadbed

Thanks for the answer. This little issue cost me 2 days. – Haplography

on.('file',(fieldname, file, filename, encoding, mimetype) => {...} is the function really providing those arguments? – Murguia

Is this answer still relevant? – Cottrill

ok now i understood that the file event fired for a file, i was using it for single file thinking it triggers for multiple chunks. – Leukorrhea

Answer 2

Thanks to the answers above I've built a npm module for this (github)

It works with google cloud functions, just install it (npm install --save express-multipart-file-parser) and use it like this:

const fileMiddleware = require('express-multipart-file-parser')

...
app.use(fileMiddleware)
...

app.post('/file', (req, res) => {
  const {
    fieldname,
    filename,
    encoding,
    mimetype,
    buffer,
  } = req.files[0]
  ...
})

Harker

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes – Norrisnorrv

Added code example, just keeping the links as references – Comradery

Can you porvide more examples on your github rpo please? – Pontus

@S.Bozzoni, about what exactly?? – Comradery

Some examples of uses, i looked for it but i am not a node expert. It seems really easy compared to busboy but i would have appreciate some example, like multiple files handling and saving. Just for node neofites. Thank you. – Pontus

This is the easiest way out here. THANKS a lot for that little gem of npm module. Works perfect! – Phototherapy

@CristóvãoTrevisan great work on your module! Personally I'd like to see an example of a valid request, made to the documented POST endpoint, in the repo as well. Also a mention of why exactly this fixes issues with Firebase would be a helpful addition as well. – Macerate

I tried this and the req.files[0] has: fieldname, originalname, encoding, mimetype, buffer – Lesotho

It works locally perfect - however deployed function in firebase not finding file - req.files is undefind anythought ? – Mcadoo

Answer 3

I was able to combine both Brian's and Doug's response. Here's my middleware that end's up mimicking the req.files in multer so no breaking changes to the rest of your code.

module.exports = (path, app) => {
app.use(bodyParser.json())
app.use(bodyParser.urlencoded({ extended: true }))
app.use((req, res, next) => {
    if(req.rawBody === undefined && req.method === 'POST' && req.headers['content-type'].startsWith('multipart/form-data')){
        getRawBody(req, {
            length: req.headers['content-length'],
            limit: '10mb',
            encoding: contentType.parse(req).parameters.charset
        }, function(err, string){
            if (err) return next(err)
            req.rawBody = string
            next()
        })
    } else {
        next()
    }
})

app.use((req, res, next) => {
    if (req.method === 'POST' && req.headers['content-type'].startsWith('multipart/form-data')) {
        const busboy = new Busboy({ headers: req.headers })
        let fileBuffer = new Buffer('')
        req.files = {
            file: []
        }

        busboy.on('field', (fieldname, value) => {
            req.body[fieldname] = value
        })

        busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
            file.on('data', (data) => {
                fileBuffer = Buffer.concat([fileBuffer, data])
            })

            file.on('end', () => {
                const file_object = {
                    fieldname,
                    'originalname': filename,
                    encoding,
                    mimetype,
                    buffer: fileBuffer
                }

                req.files.file.push(file_object)
            })
        })

        busboy.on('finish', () => {
            next()
        })


        busboy.end(req.rawBody)
        req.pipe(busboy)
    } else {
        next()
    }
})}

Beadle

Thanks a lot for this answer! Really appreciate it. – Tess

This method throws away all string attribute passed by multipart form – Lemar

Yup, just added a fix to the example. It basically adds busboy.on('field') and saves it to the OG req.body. Thanks for the feedback! @Lemar – Beadle

Thanks for this, I've posted it to npm, check my answer bellow – Comradery

Adding this code deletes all my other cloud functions. Also just wondering, what URL path would one post to with this enabled? – Bushing

for newby expressJS users, How should we add that code into our expressJS app ? furthermore it'd be cleaner if you added requieres on top, besides that why path is not being used ?. Anyway I use this approach this way: const uploadFiles = require("./upload-file/upload-file-middleware"); uploadFiles('',app); – Taillight

Answer 4

I have been suffering from the same problem for a few days, turns out that firebase team has put the raw body of multipart/form-data into req.body with their middleware. If you try console.log(req.body.toString()) BEFORE processing your request with multer, you will see your data. As multer creates a new req.body object which is overriding the resulting req, the data is gone and all we can get is an empty req.body. Hopefully the firebase team could correct this soon.

Investment

See my answer for more details. – Settles

Answer 5

Cloud functions pre-processes the request object before passing it on further. As such the original multer middleware doesn't work. Furthermore, using busboy is too low level and you need to take care of everything on your own which isn't ideal. Instead you can use a forked version of multer middleware for processing multipart/form-data on cloud functions.

Here's what you can do.

1. Install the fork

npm install --save emadalam/multer#master

2. Use startProcessing configuration for custom handling of req.rawBody added by cloud functions.

const express = require('express')
const multer = require('multer')

const SIZE_LIMIT = 10 * 1024 * 1024 // 10MB
const app = express()

const multipartFormDataParser = multer({
  storage: multer.memoryStorage(),
  // increase size limit if needed
  limits: {fieldSize: SIZE_LIMIT},
  // support firebase cloud functions
  // the multipart form-data request object is pre-processed by the cloud functions
  // currently the `multer` library doesn't natively support this behaviour
  // as such, a custom fork is maintained to enable this by adding `startProcessing`
  // https://github.com/emadalam/multer
  startProcessing(req, busboy) {
    req.rawBody ? busboy.end(req.rawBody) : req.pipe(busboy)
  },
})

app.post('/some_route', multipartFormDataParser.any(), function (req, res, next) {
  // req.files is array of uploaded files
  // req.body will contain the text fields
})

Retiary

Is this still, working? How can I install the dependency? I can't find the npm package for it – Jacobba

Answer 6

To add to the official Cloud Function team answer, you can emulate this behavior locally by doing the following (add this middleware higher than the busboy code they posted, obviously)

const getRawBody = require('raw-body');
const contentType = require('content-type');

app.use(function(req, res, next){
    if(req.rawBody === undefined && req.method === 'POST' && req.headers['content-type'] !== undefined && req.headers['content-type'].startsWith('multipart/form-data')){
        getRawBody(req, {
            length: req.headers['content-length'],
            limit: '10mb',
            encoding: contentType.parse(req).parameters.charset
        }, function(err, string){
            if (err) return next(err);
            req.rawBody = string;
            next();
        });
    }
    else{
        next();
    }
});

Yttrium

Answer 7

I ran into this issue today, check here for more details on how to handle files on google cloud (basically you don't need multer).

Here is a middleware I use to extract files. This will keep all your files on request.files and other form fields on request.body for all POST with multipart/form-data content type. It will leave everything else the same for your other middlewares to handle.

// multiparts.js
const { createWriteStream } = require('fs')
const { tmpdir } = require('os')
const { join } = require('path')
const BusBoy = require('busboy')

exports.extractFiles = async(req, res, next) => {
  const multipart = req.method === 'POST' && req.headers['content-type'].startsWith('multipart/form-data')
  if (!multipart) return next()
  //
  const busboy = new BusBoy({ headers: req.headers })
  const incomingFields = {}
  const incomingFiles = {}
  const writes = []
  // Process fields
  busboy.on('field', (name, value) => {
    try {
      // This will keep a field created like so form.append('product', JSON.stringify(product)) intact
      incomingFields[name] = JSON.parse(value)
    } catch (e) {
      // Numbers will still be strings here (i.e 1 will be '1')
      incomingFields[name] = value
    }
  })
  // Process files
  busboy.on('file', (field, file, filename, encoding, contentType) => {
    // Doing this to not have to deal with duplicate file names
    // (i.e. TIMESTAMP-originalName. Hmm what are the odds that I'll still have dups?)
    const path = join(tmpdir(), `${(new Date()).toISOString()}-${filename}`)
    // NOTE: Multiple files could have same fieldname (which is y I'm using arrays here)
    incomingFiles[field] = incomingFiles[field] || []
    incomingFiles[field].push({ path, encoding, contentType })
    //
    const writeStream = createWriteStream(path)
    //
    writes.push(new Promise((resolve, reject) => {
      file.on('end', () => { writeStream.end() })
      writeStream.on('finish', resolve)
      writeStream.on('error', reject)
    }))
    //
    file.pipe(writeStream)
  })
  //
  busboy.on('finish', async () => {
    await Promise.all(writes)
    req.files = incomingFiles
    req.body = incomingFields
    next()
  })
  busboy.end(req.rawBody)
}

And now in your function, make sure that this is the first middleware you use.

// index.js
const { onRequest } = require('firebase-functions').https
const bodyParser = require('body-parser')
const express = require('express')
const cors = require('cors')
const app = express()
// First middleware I'm adding
const { extractFiles } = require('./multiparts')
app.use(extractFiles)
app.use(bodyParser.urlencoded({ extended: true }))
app.use(bodyParser.json())
app.use(cors({ origin: true }))

app.use((req) => console.log(req.originalUrl))

exports.MyFunction = onRequest(app);

Phelgon

Answer 8

I fixed some bugs G. Rodriguez's response. I add 'field' and 'finish' event for Busboy, and do next() in 'finish' event. This is work for me. As follow:

    module.exports = (path, app) => {
    app.use(bodyParser.json())
    app.use(bodyParser.urlencoded({ extended: true }))
    app.use((req, res, next) => {
        if(req.rawBody === undefined && req.method === 'POST' && req.headers['content-type'].startsWith('multipart/form-data')){
            getRawBody(req, {
                length: req.headers['content-length'],
                limit: '10mb',
                encoding: contentType.parse(req).parameters.charset
            }, function(err, string){
                if (err) return next(err)
                req.rawBody = string
                next()
            })
        } else {
            next()
        }
    })

    app.use((req, res, next) => {
        if (req.method === 'POST' && req.headers['content-type'].startsWith('multipart/form-data')) {
            const busboy = new Busboy({ headers: req.headers })
            let fileBuffer = new Buffer('')
            req.files = {
                file: []
            }

            busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
                file.on('data', (data) => {
                    fileBuffer = Buffer.concat([fileBuffer, data])
                })

                file.on('end', () => {
                    const file_object = {
                        fieldname,
                        'originalname': filename,
                        encoding,
                        mimetype,
                        buffer: fileBuffer
                    }

                    req.files.file.push(file_object)
                })
            })

            busboy.on('field', function(fieldname, val, fieldnameTruncated, valTruncated) {
              console.log('Field [' + fieldname + ']: value: ' + inspect(val));
            });

            busboy.on('finish', function() {
              next()
            });

            busboy.end(req.rawBody)
            req.pipe(busboy);
        } else {
            next()
        }
    })}

Romilda

This method throws away all string attribute passed by multipart form – Lemar

Answer 9

Thanks for everyone's help on this thread. I wasted a whole day trying every possible combination and all these different libraries... only to discover this after exhausting all other options.

Combined some of the above solutions to create a TypeScript and middleware capable script here:

https://gist.github.com/jasonbyrne/8dcd15701f686a4703a72f13e3f800c0

Heartsome

Answer 10

If you just want to get a single uploaded file from the request, use busboy to get the file as a readable stream:

const express = require('express')
const Busboy = require('busboy')

express().post('/', (req, res) => {
  const busboy = new Busboy({ headers: req.headers })

  busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
    // Do something with `file`, e.g. pipe it to an output stream.
    // file.pipe(fs.createWriteStream('upload.pdf')
  })

  // The original input was moved to `req.rawBody`
  busboy.write(req.rawBody)
})

Improvisator

Answer 11

Note that, on top of using Busboy on the server and parsing the rawReq, you may also need to add the following config to your Axios request:

{ headers: { 'content-type': `multipart/form-data; boundary=${formData._boundary}` }};

If you only specify the content-type and not the boundary you get a Boundary not found error on the server. If you remove the headers altogether, instead, Busboy won't parse the fields properly. See: Firebase Cloud Functions and Busboy not parsing fields or files

Gingerly

Answer 12

I experience the same issue when i deployed my app using firebase function. I was using multer to upload image to amazon s3. I resolve this issue by using the above npm https://mcmap.net/q/363070/-how-to-perform-an-http-file-upload-using-express-on-cloud-functions-for-firebase-multer-busboy created by Cristóvão.

  const { mimetype, buffer, } = req.files[0]

  let s3bucket = new aws.S3({
     accessKeyId: functions.config().aws.access_key,
     secretAccessKey: functions.config().aws.secret_key,
  });

  const config = {
     Bucket: functions.config().aws.bucket_name,
     ContentType: mimetype,
     ACL: 'public-read',
     Key: Date.now().toString(),
     Body: buffer,    
   }

   s3bucket.upload(config, (err, data) => {
     if(err) console.log(err)

     req.file = data;
     next()
  })

Note that this is for a single file image upload. The next middleware will have the returned object from s3

{ 
  ETag: '"cacd6d406f891e216f9946911a69aac5"',
  Location:'https://react-significant.s3.us-west1.amazonaws.com/posts/1567282665593',
  key: 'posts/1567282665593',
  Key: 'posts/1567282665593',
  Bucket: 'react-significant' 
}

In this case, you might need the Location url before you save your data in the db.

Canonical

Answer 13

I've tried Dougs answer, however the finish was never fired, so i tweaked the code a little bit and got this which works for me:

// It's very crucial that the file name matches the name attribute in your html
app.post('/', (req, res) => {
  const busboy = new Busboy({ headers: req.headers })
  // This object will accumulate all the uploaded files, keyed by their name
  const uploads = {}

  // This callback will be invoked for each file uploaded
  busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
    console.log(`File [${fieldname}] filename: ${filename}, encoding: ${encoding}, mimetype: ${mimetype}`)
    // Note that os.tmpdir() is an in-memory file system, so should only
    // be used for files small enough to fit in memory.
    const filepath = path.join(os.tmpdir(), filename)
    uploads[fieldname] = { file: filepath }
    console.log(`Saving '${fieldname}' to ${filepath}`)
    const stream = fs.createWriteStream(filepath)
    stream.on('open', () => file.pipe(stream))
  })

  // This callback will be invoked after all uploaded files are saved.
  busboy.on('finish', () => {
    console.log('look im firing!')
// Do whatever you want here
    res.end()
  })

  // The raw bytes of the upload will be in req.rawBody.  Send it to busboy, and get
  // a callback when it's finished.
  busboy.end(req.rawBody)
})

Inbound

Answer 14

Here is my version for typescript:

import { FileInfo, } from 'busboy';
import Busboy from 'busboy';


app.post('/images', (req, resp, next) => {
  const busboy = Busboy({ headers: req.headers });
  const allFiles: FileInfo[] = [];

  busboy.on('file', (fieldname: string, file: any, fileInfo: FileInfo) => {
    const { filename, encoding, mimeType } = fileInfo;
    console.log(`Fiild ${fieldname}, File: ${file}, filename: ${filename}, encoding: ${encoding}, mimetype: ${mimeType}`);
    allFiles.push(fileInfo);
    file.on('data', (data: Uint8Array) => {
      console.log(`File got ${data.length} bytes`);
    })
  });

  busboy.on('finish', () => {
    resp.send(allFiles);
  });

  busboy.on('error', () => {
    resp.status(400);
  });

  busboy.end((req as any).rawBody);

});

app can be either express app or express router..

Demonolater

I've tried using this as a middleware like: app.post( "/files/upload", BusboyMiddleware.filesUpload(), Controller.handlePostFileUploads, ); but I got a 400 each time, at Multipart.eval – Jacobba

you could log the error in on error cb – Demonolater