how to make formidable not save to var/folders on nodejs and express app

Question

I'm using formidable to parse incoming files and store them on AWS S3

When I was debugging the code I found out that formidable is first saving it to disk at /var/folders/ and overtime some unnecessary files are stacked up on disk which could lead to a big problem later.

It's very silly of me using a code without fully understanding it and now

I have to figure out how to either remove the parsed file after saving it to S3 or save it to s3 without storing it in disk. But the question is how do I do it?

I would appreciate if someone could point me in the right direction

this is how i handle the files:

import formidable, { Files, Fields } from 'formidable';

const form = new formidable.IncomingForm();
form.parse(req, async (err: any, fields: Fields, files: Files) => {
let uploadUrl = await util
            .uploadToS3({
                file: files.uploadFile,
                pathName: 'myPathName/inS3',
                fileKeyName: 'file',
            })
            .catch((err) => console.log('S3 error =>', err));
}

Symbolics

Answer 1

This is how i solved this problem:

When I parse incoming form-multipart data I have access to all the details of the files. Because it's already parsed and saved to local disk on the server/my computer. So using the path variable given to me by formidable I unlink/remove that file using node's built-in fs.unlink function. Of course I remove the file after saving it to AWS S3.

This is the code:

import fs from 'fs';
import formidable, { Files, Fields } from 'formidable';

const form = new formidable.IncomingForm();
form.multiples = true;

form.parse(req, async (err: any, fields: Fields, files: Files) => {
   const pathArray = [];
   try {
      const s3Url = await util.uploadToS3(files);
      // do something with the s3Url
      pathArray.push(files.uploadFileName.path);
   } catch(error) {
      console.log(error)
   } finally {
      pathArray.forEach((element: string) => {
         fs.unlink(element, (err: any) => {
            if (err) console.error('error:',err);
            });
        });
   }
})

I also found a solution which you can take a look at here but due to the architecture if found it slightly hard to implement without changing my original code (or let's just say I didn't fully understand the given implementation)

Symbolics

I see this is an old question but is there something official about this instead of "hacking"? Its so annoying to see that every file is saved in some completely "hidden" folder without knowing. – Palestra

@Palestra I came across a new npm package the other day called pechkin. You might consider taking a look at it and play with it. Speaking about official, I think these days the way saving files to cloud storage is done using presigned ulrs rather than involving backend servers. Presigned urls help decrease upload time as well since you directly save it to the cloud – Symbolics

Answer 2

I think i found it. According to the docs see options.fileWriteStreamHandler, "you need to have a function that will return an instance of a Writable stream that will receive the uploaded file data. With this option, you can have any custom behavior regarding where the uploaded file data will be streamed for. If you are looking to write the file uploaded in other types of cloud storages (AWS S3, Azure blob storage, Google cloud storage) or private file storage, this is the option you're looking for. When this option is defined the default behavior of writing the file in the host machine file system is lost."

const form = formidable({
    fileWriteStreamHandler: someFunction,
  });

EDIT: My whole code

import formidable from "formidable";
import { Writable } from "stream";
import { Buffer } from "buffer";
import { v4 as uuidv4 } from "uuid";


export const config = {
  api: {
    bodyParser: false,
  },
};

const formidableConfig = {
  keepExtensions: true,
  maxFileSize: 10_000_000,
  maxFieldsSize: 10_000_000,
  maxFields: 2,
  allowEmptyFiles: false,
  multiples: false,
};

// promisify formidable
function formidablePromise(req, opts) {
  return new Promise((accept, reject) => {
    const form = formidable(opts);

    form.parse(req, (err, fields, files) => {
      if (err) {
        return reject(err);
      }
      return accept({ fields, files });
    });
  });
}

const fileConsumer = (acc) => {
  const writable = new Writable({
    write: (chunk, _enc, next) => {
      acc.push(chunk);
      next();
    },
  });

  return writable;
};

// inside the handler

export default async function handler(req, res) {
  const token = uuidv4();
  try {
    const chunks = [];
    const { fields, files } = await formidablePromise(req, {
      ...formidableConfig,
      // consume this, otherwise formidable tries to save the file to disk
      fileWriteStreamHandler: () => fileConsumer(chunks),
    });

    // do something with the files
  
    const contents = Buffer.concat(chunks);

    const bucketRef = storage.bucket("your bucket");

    const file = bucketRef.file(files.mediaFile.originalFilename);

    await file
      .save(contents, {
        public: true,
        metadata: {
          contentType: files.mediaFile.mimetype,
          metadata: { firebaseStorageDownloadTokens: token },
        },
      })
      .then(() => {
        file.getMetadata().then((data) => {
        
          const fileName = data[0].name;

          const media_path = `https://firebasestorage.googleapis.com/v0/b/${bucketRef?.id}/o/${fileName}?alt=media&token=${token}`;
          console.log("File link", media_path);
          
        });
      });
  } catch (e) {
    // handle errors
    console.log("ERR PREJ ...", e);
  }
}

Palestra

Yes, this must be the way to go instead of instead of the accepted answer which is a hacky workaround. But at the time neither did I understand what streams are nor formidable had this option. I did not take a look at what exactly this option does. Does it save still files to the file system and then feeds the stream to cloud storage or does it directly steams to the cloud storage? If it's the latter this might very well be the accepted answer – Symbolics

Yes, it created a buffer from the file and streams it directly to the cloud. I tested it and no file was saved in the local machine. Check my updated answer for whole code. – Palestra

Is it possible to pipe from stream to stream directly without buffering? – Symbolics

I believe it is. TBH didn't try that. – Palestra