creating confidence area for normally distributed scatterplot in ggplot2 and R

Question

I have some data, say (in reality, I have a large amount of data):

x   y
0.1 0.267
0.2 0.254
0.3 0.182
0.4 0.173
0.5 0.121
0.6 0.089
0.7 0.070
0.8 0.056
0.9 0.031

This data roughly follows a trend curve that I plot with ggplot stat_smooth(). stat_smooth creates a grey area that it is 95% confident that the trendline will fall inside.

What I wish to do instead is create a grey area around where the trendline will be is create a grey area around where the data points will fall with 95% confidence assuming a normal distribution around the trendline.

Another way of saying it, is I want a smooth_lines connecting the tops and bottoms of the error bars and to shade inbetween in ggplot. How should I go about doing this?

Thanks.

Mica

Answer 1

What you are looking for is essentially called a prediction interval. Here is one way to do it in ggplot2

library(ggplot2)

# RUN REGRESSION AND APPEND PREDICTION INTERVALS
lm_fit  = lm(total_bill ~ tip, data = tips)
tips_with_pred = data.frame(tips, predict(lm_fit, interval = 'prediction'))

# PLOT WITH REGRESSION LINE, CONFIDENCE INTERVAL AND PREDICTION INTERVAL
p0 <- ggplot(tips_with_pred, aes(x = tip, y = total_bill)) + 
  geom_point() +
  geom_smooth(method = 'lm', aes(fill = 'confidence'), alpha = 0.5) +
  geom_ribbon(aes(y = fit, ymin = lwr, ymax = upr, fill = 'prediction'),
    alpha = 0.2) +
  scale_fill_manual('Interval', values = c('green', 'blue')) +
  opts(legend.position = c(0.20, 0.85))

Libidinous

interesting data... There are many generous people. I thought, though, x and y axis should be reversed, no? – Gastronomy

yes. i just cobbled up something quickly to illustrate the construction of the two intervals. – Libidinous

Beautiful. I think this is the post that's finally convinced me to buckle down and learn a 3rd R graphical system. – Hygiene