fread to read top n rows from a large file

Question

I am getting below error when reading first n rows from a big file(around 50 GB) using fread. Looks like a memory issue. I tried to use nrows=1000 . But no luck. Using linux

file ok but could not memory map it. This is a 64bit process. There is probably not enough contiguous virtual memory available.

Can this below code be replaced with read.csv with all options as used below? Does it help?

  rdata<- fread(
      file=csvfile, sep= "|", header=FALSE, col.names= colsinfile,
    select= colstoselect, key = "keycolname", na.strings= c("", "NA")
    , nrows= 500
  )

Colossal

What if you replace csvfile with paste('head -n 500', csvfile)? – Ingalls

@Ingalls : got an error File 'head -n 500 /csvfile' doesnt exist – Colossal

The argument should finally looks like input = "head -n 500 /path/to/csvfile". Please use the input argument rather than file argument to allow shell commands. I do not have a file that large to test. I hope this works. – Ingalls

@Ingalls : thats awsome. when used with input it works!.. You should put this as answer – Colossal

Answer 1

Another workaround is to fetch the first 500 lines with shell command:

rdata<- fread(
    cmd = paste('head -n 500', csvfile),
    sep= "|", header=FALSE, col.names= colsinfile,
    select= colstoselect, key = "keycolname", na.strings= c("", "NA")
)

I don't known why nrows doesn't work, though.

Ingalls

Newer versions (somewhere between 3.4.0 and 3.6.0) recommend cmd = instead of input =. – Certify

Answer 2

Perhaps this would help you:

processFile = function(filepath) {
con = file(filepath, "r")
while ( TRUE ) {
line = readLines(con, n = 1)
if ( length(line) == 0 ) {
  break
}
print(line)
}
close(con)
}

see reading a text file in R line by line.. In your case you'd probably want to replace the while ( TRUE ) by for(i in 1:1000)

Pentad