unexpected results: microbenchmark

Question

I've always been bugged a bit by the lack of accuracy I see bench marking with system.time and rbenchmark (in that the precision of the timing may be lacking) and saw Hadley reference the microbenchmark package recently. So I decided to give it a whirl as seen below. I pitted mean against f <- function(x) {sum(x)/length(x)} and expected mean to do way better than f but the results, as I understand them, do not indicate this to be true.

1. Am I misunderstanding the results?
2. Is f actually faster than mean?
3. Is microbenchmark still in a beta phase and this needs to be ironed out?

I'm running R2.15 on a win 7 machine (as microbenchmark does timings differently depending on your OS).

The Results

Unit: microseconds
     expr    min     lq median     uq    max
1    f(x) 19.130 20.529 20.529 20.996 286.00
2 mean(x) 28.927 29.860 30.327 30.327 672.31

The Code

library(microbenchmark)

x <- 1:10000
f <- function(x) {sum(x)/length(x)}
mean(x)

res <- microbenchmark(
    mean(x), 
    f(x), 
times=1000L)

print(res)
boxplot(res)

Motorcycle

I like microbenchmark. If you're doing more than one or two results, plotting can help greatly but the default output is a little on the ugly side. I wrote an autoplot function for ggplot2 which may show up in one of these releases (check github in the meantime). Examples: https://mcmap.net/q/1164748/-running-functions-in-r-multiple-times-during-benchmarking – Gorilla

This may explain it radfordneal.wordpress.com/2014/02/02/… – Zhukov

Probably not as all stats for the f were lower and a scatterplot indicated this as well. joran nailed this one. – Motorcycle

Answer 1

I could be wrong, but this doesn't seem all that surprising to me. Before mean.default can call .Internal(mean(x)) it has to check 3 if statements, calculate the length of x, and then check another if statement. And the difference in times it fairly small.

Calling .Internal(mean(x) directly is slightly faster still:

library(microbenchmark)

x <- 1:10000
f1 <- function(x) {sum(x)/length(x)}
f2 <- function(x) {.Internal(mean(x))}

res <- microbenchmark(
    mean(x), 
    f1(x),
    f2(x), 
times=1000L)

print(res)

Unit: microseconds
     expr    min      lq  median      uq     max
1   f1(x) 32.195 32.4605 32.8850 33.4645 106.997
2   f2(x) 21.840 22.0580 22.2015 22.6270  55.316
3 mean(x) 35.393 35.9840 36.1860 36.4420  91.203

Beatrizbeattie

Thanks. I made an assumption and didn't check it. A better check of microbenchmarking would be sum against a Reduce version: sum2 <- function(x) Reduce("+", x) which yields the expected results. I think I'm going to like microbenchmark – Motorcycle

Just ran across this in unrelated readings. Apparently mean is somewhat notorious: lookingatdata.blogspot.se/2011/04/… – Gorilla

Answer 2

I think you'll find that if you bump up the size of X by a factor of 10 you'll see more consistent results. To be honest I'd be surprised if you really can get micro-second timing accuracy on a computer with a multi-tasking operating system.

You might also consider:

• Are you running on a laptop or a machine which has automatic CPU frequency scaling?
• Warmup?
• Pinning your process to one core.

Stubstad

Part of the beauty of microbenchmark is that it runs things a lot of times so you can see how the results vary, so I'd guess that what's running in the background has no effect on the results in expectation. – Gorilla

@gsk3 have a look at the output times, and you'll see a long tail, so something like... hist(res[res$expr=='f(x)','time']/length(x), breaks=90) – Stubstad

@gsk3 it depends what the question is I suppose. How fast can it go? or How fast will it go on "average"? :) – Stubstad