CSV File Processing with Nestjs and Papa Parse

Question

I am trying to process a CSV file in NestJS using Multer and Papa Parse. I do not want to store the file locally. I just want to parse CSV files to extract some information.

However, I am unable to process it, I have tried two different ways. In the first one, I passed the file buffer to Papa.parse function. However, I get the error: ReferenceError: FileReaderSync is not defined

@Post('1')
@UseInterceptors(
    FileInterceptor('file', {})
)
async uploadFile(@UploadedFile() file: Express.Multer.File ){
    const csvData = papa.parse(file.buffer, {
        header: false,
        worker: true,
        delimiter: ",",
        step: function (row){
            console.log("Row: ", row.data);
        }
      });
}

So tried calling the readFileSync() as shown below, but this time I got the error, ERROR [ExceptionsHandler] ENAMETOOLONG: name too long, open

@Post('2')
@UseInterceptors(
    FileInterceptor('file', {})
)
async uploadFile(@UploadedFile() file: Express.Multer.File ){
    const $file =   readFileSync(file.buffer);
    const csvData = papa.parse($file, {
        header: false,
        worker: true,
        delimiter: ",",
        step: function (row){
            console.log("Row: ", row.data);
        }
      });
}

will appreciate any help to resolve this issue.

Vibratile

github.com/mholt/PapaParse#papa-parse-for-node and https://mcmap.net/q/131274/-converting-a-buffer-into-a-readablestream-in-node-js – Swallowtail

Answer 1

As pointed out by @skink, the file buffer needs to be converted to stream before it can be used by papa parse.

const { Readable } = require('stream');

And updated the function, where converting the file.buffer to stream before calling parse()

@Post('1')
@UseInterceptors(
    FileInterceptor('file', {})
)
async uploadFile(@UploadedFile() file: Express.Multer.File ){
    const stream = Readable.from(file.buffer);
    const csvData = papa.parse(stream, {
        header: false,
        worker: true,
        delimiter: ",",
        step: function (row){
            console.log("Row: ", row.data);
        }
      });
}

Vibratile

Answer 2

As mentioned by @Adrian Mian, you need to convert the file.buffer to stream before calling parse() from papaparse.

In addition, depending on your architecture, if you need to pass the file to another service (or micro-service), it may be useful to first convert the file buffer to base64.

In your controller:

import { Controller, Post, Body, UseInterceptors, UploadedFile } from '@nestjs/common'
import { FileInterceptor } from '@nestjs/platform-express'
import { ApiBody, ApiConsumes, ApiOperation } from '@nestjs/swagger'

@ApiOperation({
    summary: 'Import data by uploading a CSV file.',
  })
@ApiConsumes('multipart/form-data')
@ApiBody({
    schema: {
      type: 'object',
      properties: {
        file: {
          type: 'string',
          format: 'binary',
        },
      },
    },
  })
  @Post('import')
  @UseInterceptors(FileInterceptor('file'))
  importUsers(@UploadedFile() file: Express.Multer.File) {
    return this.importService.importData(file.buffer.toString('base64')
  }

importService:

import { Readable } from 'stream'

async importData(fileBufferInBase64: string) {
  const buffer = Buffer.from(fileBufferInBase64, 'base64')
  const dataStream = Readable.from(buffer)
  const parsedCsv = parse(dataStream, {
     header: true,
     skipEmptyLines: true,
     complete: (results) => {
        console.log('results:', results)
     },
  })
}

Jemadar