I am trying to process a CSV file in NestJS using Multer and Papa Parse. I do not want to store the file locally. I just want to parse CSV files to extract some information. However, I am unable to...

I have a 10^7 lines file, in which I want to choose 1/100 of lines randomly from the file. This is the AWK code I have, but it slurps all the file content before hand. My PC memory cannot handle su...

Using GNU parallel: http://www.gnu.org/software/parallel/ I have a program that takes two arguments, e.g. $ ./prog file1 file2 $ ./prog file2 file3 ... $ ./prog file23456 file23457 I'm using a ...

I have a 10 GB log file in a particular format, I want to process this file line by line and then write the output to other file after applying some transformations. I am using node for this operat...

So I have this data frame from a bed file called input.bed: V1 V2 V3 V4 1 chr1 11323785 11617177 TF1 2 chr1 12645605 13926923 TF2 3 chr1 14750216 15119039 TF3 4 chr1 18102157 19080189 TF1 5 ...

I have a file in which I'd like to iterate without processing in any sort the current line. What I am looking for is the best way to go to a determined line of a text file. For example, storing the...

How can I effectively read from a large file and write bulk data into a file using the Java NIO framework. I'm working with ByteBuffer and FileChannel and had tried something like below: public s...

I need to develop an application that will process csv files as soon as the files are created in a predefined directory. Huge number of incoming files is expected. I have seen applications using A...

I am trying to append the name of a folder to all filenames within that folder. I have to loop through a parent folder that contain sub folders. I have to do this in Python and not a bat file. Exa...

I have a spring batch project that reads a huge zip file containing more than 100.000 xml files. I am using MultiResourcePartitioner, and I have a Memory issue and my batch fails with java.lang....

I am processing text files 60GB or larger. The files are seperated into a header section of variable length and a data section. I have three functions: head? a predicate to distinguish header lin...

I have this code in my Web API app to write to a CSV file: private void SaveToCSV(InventoryItem invItem, string dbContext) { string csvHeader = "id,pack_size,description,vendor_id,department,subd...

I'm trying to process a very large unicode text file (6GB+). What I want is to count the frequency of each unique word. I use a strict Data.Map to keep track of the counts of each word as I travers...

Does anyone know of any parallel equivalent of java Files.walkFileTree or something similar? It can be Java or Scala library.

I am convinced it is something simply syntactic - I however can not figure out why my code: import os from collections import Counter d = {} for filename in os.listdir('testfilefolder'): f = open...

I have a crawler program which logs some files. Sometimes on the server, some error happens and the crawler creates massive log files which are somehow impossible to parse. For that reason, I wante...

We want to create a relative simple document storage but there are some requirements. My idea was, that a file is scanned and handled by a separate tool/daemon when it arrives at storage immediatel...

I have to process some data by combining two different files. Both of them have two columns that would form a primary key that I can use to match them side-by-side. The files in questions are huge ...

Following up on this question, I need to get exactly n lines at random out of a file (or stdin). This would be similar to head or tail, except I want some from the middle. Now, other than looping ...