Memory profiling in R - tools for summarizing

Question

R has some tools for memory profiling, like Rprofmem(), Rprof() with option "memory.profiling=TRUE" and tracemem(). The last one can only be used on objects, and hence is useful to follow how many times an object is copied, but doesn't give an overview on a function basis. Rprofmem should be able to do that, but the output of even the simplest function call like lm() gives over 500 lines of log. I tried to figure out what Rprof("somefile.log",memory.profile=T) actually does, but I don't think I really get it.

The last I could find was this message of Thomas Lumley, saying that, and I quote :

I do not yet have tools to summarize the output.

This was in 2006. Any chance there are options for some nice summaries now, based on either Rprofmem(), the mysterious output of Rprof() with memory.profile set TRUE or any other tool?

Pneumonectomy

Already looking forward to your RprofmemSummary package :) – Eicher

@Dirk I reckon that's a "Good luck, poor lad..." :) – Pneumonectomy

I'd to add this capability to profr. Hoping to find an interested student one day. It could be a good google summer of code project if you wanted to write it up. I'd be happy to co-mentor. – Incandescent

@Incandescent : I currently have no time to start with this, but if you didn't find an interested student by the summer, you can take me up on the challenge. Thx for the offer. – Pneumonectomy

Hi everyone, I just read this..is there any advance since Mar 2011 ? – Rwanda

I didn't see any progress, but I haven't been checking either. Maybe a good thing to pick up when I finished the rest of the projects I still have to do... – Pneumonectomy

from developer.r-project.org/memory-profiling.html Rprof has an option memory.profiling. ... This is available only on Unix at the moment. – Hulen

Answer 1

profvis looks like the the solution to this question.

It generates an interactive .html file (using htmlwidgets) showing the profiling of your code.

The introduction vignette is a good guide on its capability.

Taking directly from the introduction, you would use it like this:

devtools::install_github("rstudio/profvis")
library(profvis)

# Generate data
times <- 4e5
cols <- 150
data <- as.data.frame(x = matrix(rnorm(times * cols, mean = 5), ncol = cols))
data <- cbind(id = paste0("g", seq_len(times)), data)
profvis({
    data1 <- data   # Store in another variable for this run

    # Get column means
    means <- apply(data1[, names(data1) != "id"], 2, mean)

    # Subtract mean from each column
    for (i in seq_along(means)) {
        data1[, names(data1) != "id"][, i] <- data1[, names(data1) != "id"][, i] - means[i]
    }
}, height = "400px")

Which gives

Blowzy

thx pal, didn't know about this one yet. Valuable tool for sure. – Pneumonectomy

@JorisMeys - no worries, only just found it myself. Worth noting it's still in development so it's a relatively new package. – Blowzy

Answer 2

Check out profr -- it seems like exactly what you're looking for.

Bakery