Use node to create a tarball from memory buffers (non disk) and store result in memory

Question

Say i have the two buffers:

const bufferFile1 = Buffer.from('Hello World!', 'utf-8')
const bufferFile2 = Buffer.from('Hello Again World!', 'utf-8')

How can i create tarball file buffer/stream/blob (not written to disk) where the the two buffers above should be stored as two files in the tarball. I would like to (be able to) pipe the tarball as a response to a request.

I have looked into using the tar package. But this solution requires paths instead of in memory streams/buffers.

Is it at all possible to accomplish this?

PS: I have very little experience with in memory file handling.

---

The overall project is to create an api endpoint

• takes in parameters either as POST body or url query parameters.
• creates a set of in memory files with content based on input
• creates a tar.gz file based on these files
• stores the tar.gz file in mongodb using GridFS

I dont really have a place to store the files temporarily (serverless environment), thats why i want the solution to be completely in memory.

Individualism

it seems that this package might be useful: npmjs.com/package/tar-stream – Individualism

I've done it using github.com/archiverjs/node-archiver – Mammillary

Answer 1

I was able to get it working with archiver and axios.

archiver is a wrapper of tar-stream and zip-stream. So this is possible in a similar manner with tar-stream only, but I didn't get it working.

If archive.read() is giving null you should make sure

1. archive.read() is inside of archive.on("finish",...
2. archive.on("finish",... is called before archive.finalize()

Here is my full code snippet:

const axios = require("axios");
const archiver = require("archiver");

const archive = archiver("tar", { gzip: false, store: false });

// data is a string representation of the text.
archive.append(data, { name: "main.txt" });


archive.on("finish", () => {
  const blob = new Blob([archive.read()], { type: "application/x-tar" });
  const formData = new FormData();
  // FormData seems to only accept blob
  formData.append("file", blob);
  axios
    .post(url, formData, {
      // remove this if your file is not in a binary format
      responseType: 'arraybuffer'
    })
    .then((response) => {
      console.log(response.data); // the response and data 
    })
    .catch((error) => {
      console.error(error);
    });
});

// doing this after registering the events or else it will the data might not be present
archive.finalize();

Cero