Programmatically extract an object from collection of RData files

Question

We work in a production environment, with large datasets assembled from API calls saved as RData files to retain the full environment and subsequent data summaries. The RData files are very large, and contain multiple dataframe objects generated with a standard analysis workflow with similar names and structures.

I'm looking for a clean way to walk through the collection of RData files, pull a named object from each, then assemble into an AllCohorts dataframe for analysis.

Kaseykasha

Answer 1

The following modification (1) allows a request for several objects and (2) avoid using an assignment. i.e., places the objects in the calling environnement

extractRData <- function(file, objects) {
  objectsNotFound <- c()
  E <- new.env()
  load(file=file, envir=E)
  for (object in objects) {
    temp <- try({
      get(object, envir=E, inherits=F)
    })
    if (substr(temp[1],1,5) == "Error") {
      objectsNotFound <- c(objectsNotFound,object)
    } else {
      eval(parse(text=paste(object," <<- temp")))
    }
  }
  return(objectsNotFound)
}

Involved

Answer 2

We found a useful solution.

1. An extractor function

 extractorRData <- function(file, object) {
      #' Function for extracting an object from a .RData file created by R's save() command
      #' Inputs: RData file, object name
      E <- new.env()
      load(file=file, envir=E)
      return(get(object, envir=E, inherits=F))
    }

2. Extract a data frame "allParams" from RData file "priorRun.RData" without loading the entire environment.

      allParams.prior <- extractorRData("priorRun.RData", "allParams")

This approach has proven to be both fast, and flexible. Useful with large data frames that would be slow to reconstruct.

Kaseykasha