Bind residuals to input dataset with missing values [duplicate]

Question

I am looking for a method to bind lm residuals to an input dataset. The method must add NA for missing residuals and the residuals should correspond to the proper row.

Sample data:

N <- 100 
Nrep <- 5 
X <- runif(N, 0, 10) 
Y <- 6 + 2*X + rnorm(N, 0, 1) 
X[ sample(which(Y < 15), Nrep) ] <- NA
df <- data.frame(X,Y)

residuals(lm(Y ~ X,data=df,na.action=na.omit))

Residuals should be bound to df.

Petuntse

Similar questions here and here. – Buffer

Answer 1

"[<-"(df, !is.na(df$X), "res", residuals(lm(Y ~ X,data=df,na.action=na.omit)))

will do the trick.

Grizel

Can you explain this? What is "[<-"? – Raver

@BrandonBertelsen The function "[<-"(x1, x2, x3, x4) is similar to x1[x2, x3] <- x4 but leaves x1 unchanged and returns a new object. – Grizel

Answer 2

Simply change the na.action to na.exclude:

residuals(lm(Y ~ X, data = df, na.action = na.exclude))

na.omit and na.exclude both do casewise deletion with respect to both predictors and criterions. They only differ in that extractor functions like residuals() or fitted() will pad their output with NAs for the omitted cases with na.exclude, thus having an output of the same length as the input variables.

(this is the best solution found here)

Buffer

This is the general solution you're looking for, the one that works with missings in any number of predictors or DV, with lm and lme4. – Hypomania

Answer 3

Using merge, or join.

N <- 100 
Nrep <- 5 
X <- runif(N, 0, 10) 
Y <- 6 + 2*X + rnorm(N, 0, 1) 
X[ sample(which(Y < 15), Nrep) ] <- NA
df <- data.frame(X,Y)

df$id <- rownames(df)

res <- residuals(lm(Y ~ X,data=df,na.action=na.omit))
tmp <- data.frame(res=res)
tmp$id <- names(res)

merge(df,tmp,by="id",sort=FALSE,all.x=TRUE)

If you need to maintain the order. Use join() from the plyr package:

library(plyr) 
join(df,tmp)

Raver

couldn't this code be simplified by merging by row names? – Buffer

There is much much simpler solution, see my answer – Buffer

Answer 4

This maybe could be solution, but, first, you do not need c() in data.frame

df <- data.frame(X,Y)
df$Res[!is.na(X)]<-residuals(lm(Y ~ X,data=df,na.action=na.omit))

Altricial

This duplicates residuals. Rather than appending NA – Raver

I've removed the c() in data.frame – Petuntse

What if Y is NA? What if another predictor variables is NA? Not very robust to this, thus probably not a way to go. – Buffer

Answer 5

N <- 100 
Nrep <- 5 
X <- runif(N, 0, 10) 
Y <- 6 + 2*X + rnorm(N, 0, 1) 
X[ sample(which(Y < 15), Nrep) ] <- NA
df <- data.frame(X,Y)

R.all=as.numeric(rep(NA,length(X)))  # numeric vector with missing values
res=residuals(lm(Y ~ X,data=df,na.action=na.omit))  
i=as.numeric(names(res)) # vector locations of non-missing residuals
R.all[i]=res;R.all     # assign residuals to their correct positions.

Training