predict.lm() in a loop. warning: prediction from a rank-deficient fit may be misleading

Question

This R code throws a warning

# Fit regression model to each cluster
y <- list() 
length(y) <- k
vars <- list() 
length(vars) <- k
f <- list()
length(f) <- k

for (i in 1:k) {
  vars[[i]] <- names(corc[[i]][corc[[i]]!= "1"])
  f[[i]]  <- as.formula(paste("Death ~", paste(vars[[i]], collapse= "+")))
  y[[i]]  <- lm(f[[i]], data=C1[[i]]) #training set
  C1[[i]] <- cbind(C1[[i]], fitted(y[[i]]))
  C2[[i]] <- cbind(C2[[i]], predict(y[[i]], C2[[i]])) #test set
}

I have a training data set (C1) and a test data set (C2). Each one has 129 variables. I did k means cluster analysis on the C1 and then split my data set based on cluster membership and created a list of different clusters (C1[[1]], C1[[2]], ..., C1[[k]]). I also assigned a cluster membership to each case in C2 and created C2[[1]],..., C2[[k]]. Then I fit a linear regression to each cluster in C1. My dependant variable is "Death". My predictors are different in each cluster and vars[[i]] (i=1,...,k) shows a list of predictors' name. I want to predict Death for each case in test data set (C2[[1]],..., C2[[k]). When I run the following code, for some of the clusters.

I got this warning:

In predict.lm(y[[i]], C2[[i]]) :
prediction from a rank-deficient fit may be misleading

I read a lot about this warning but I couldn't figure out what the issue is.

Performing

The problem is that you get rank-deficient fits. You need to find out which fits give the warning and examine them. – Hunks

Answer 1

You can inspect the predict function with body(predict.lm). There you will see this line:

if (p < ncol(X) && !(missing(newdata) || is.null(newdata))) 
    warning("prediction from a rank-deficient fit may be misleading")

This warning checks if the rank of your data matrix is at least equal to the number of parameters you want to fit. One way to invoke it is having some collinear covariates:

data <- data.frame(y=c(1,2,3,4), x1=c(1,1,2,3), x2=c(3,4,5,2), x3=c(4,2,6,0), x4=c(2,1,3,0))
data2 <- data.frame(x1=c(3,2,1,3), x2=c(3,2,1,4), x3=c(3,4,5,1), x4=c(0,0,2,3))
fit <- lm(y ~ ., data=data)

predict(fit, data2)
       1        2        3        4 
4.076087 2.826087 1.576087 4.065217 
Warning message:
In predict.lm(fit, data2) :
  prediction from a rank-deficient fit may be misleading

Notice that x3 and x4 have the same direction in data. One is the multiple of the other. This can be checked with length(fit$coefficients) > fit$rank

Another way is having more parameters than available variables:

fit2 <- lm(y ~ x1*x2*x3*x4, data=data)
predict(fit2, data2)
Warning message:
In predict.lm(fit2, data2) :
  prediction from a rank-deficient fit may be misleading

Chansoo

Thank you for your response. In cluster 2, C1[[2]] has 130 rows and I have 67 predictors. I found regression function y[[2]]. Then, I used y[[2]] to predict "Death" for all cases in C2[[2]]. C2[[2]] has only 32 rows. Is this the cause of the warning? As I have 32 cases and my regression function has 67 variables? When we use predict.lm, I assumed that we already found the function and the function will be used to predict Death for each case in C2[[2]]. So, I thought it is not important to have more cases than number of predictors. Am I right? – Performing

It is important to have more cases than variables in your model. You can try doing it while having less cases than needed, but you should have in mind that your predictions might be unreliable in that case. That is the reason R gives you a "warning" and not an error. Just to draw your attention. You should be able to get your answers and continue your work even after the warnings (they are not errors), but it would be wise to try simplifying your model. – Saprolite

Answer 2

This warning:

In predict.lm(model, test) :
  prediction from a rank-deficient fit may be misleading

Gets thrown from R's predict.lm. See: http://stat.ethz.ch/R-manual/R-devel/library/stats/html/predict.lm.html

Understand rank deficiency: Ask R to tell you the rank of a matrix:

train <- data.frame(y=c(1234, 325, 152, 403), 
                   x1=c(3538, 324, 382, 335), 
                   x2=c(2985, 323, 223, 288), 
                   x3=c(8750, 322, 123, 935))
test <- data.frame(x1=c(3538, 324, 382, 335), 
                   x2=c(2985, 323, 223, 288), 
                   x3=c(8750, 322, 123, 935))
library(Matrix)
cat(rankMatrix(train), "\n")   #prints 4
cat(rankMatrix(test), "\n")    #prints 3

A matrix that does not have "full rank" is said to be "rank deficient". A matrix is said to have full rank if its rank is either equal to its number of columns or to its number of rows (or to both).

The problem is that predict.lm will throw this warning even if your matrices are full rank (not rank deficient) because predict.lm pulls a fast one under the hood, by throwing out what it considers useless features, modifying your full rank input to be rank-deficient. It then complains about it through a warning.

Also this warning seems to be a catch-all for other situations like for example you have too many input features and your data density is too sparse and it's offering up it's opinion that predictions are brittle.

Example of passing full rank matrices, yet predict.lm still complains of rank deficiency

train <- data.frame(y=c(1,2,3,4),
                        x1=c(1,1,2,3),
                        x2=c(3,4,5,2),
                        x3=c(4,2,6,0),
                        x4=c(2,1,3,0)
                   )
test <- data.frame(x1=c(1, 2,  3,  9),
                   x2=c(3, 5,  1, 15),
                   x3=c(5, 9,  5, 22),
                   x4=c(9, 13, 2, 99))
library(Matrix)
cat(rankMatrix(train), "\n")    #prints 4, is full rank, good to go
cat(rankMatrix(test), "\n")     #prints 4, is full rank, good to go
myformula = as.formula("y ~ x1+x2+x3+x4")
model <- lm(myformula, train)
predict(model, test) 
    #Warning: prediction from a rank-deficient fit may be misleading

workaround:

Assuming predict is returning good predictions, you can ignore the warning. predict.lm offers up it's opinion given insufficient perspective and here you are.

So disable warnings on the predict step like this:

options(warn=-1)      #turn off warnings
predict(model, test)
options(warn=1)      #turn warnings back on

Leeth

I think this answer is wrong. You need to take the rank of the design matrix, not the rank of the data frame that includes a column with the response variable! The train data in the second example (with columns y through x4) is rank deficient. Try X <- model.matrix(y ~ ., train); rankMatrix(X) < ncol(X) and you will see that the design matrix is rank deficient. I believe your assertion that R will modify "your full rank input to be rank-deficient" is false. – Hazem

Answer 3

It is because, one of your dependent variables has NA for Coefficients given as output by the lm(..) function. Such a variable is making no difference to the model, often due to multicollinearity problem ie, that predictor variable is linearly dependent on other predictor variables OR because, that predictor variable is constant for all the records(rows). The best thing to do is to drop that variable from the formula in lm(..) function and do the regression again. This does not reduce the accuracy of the model. In my case,

model <- lm(Happiness.Score ~ Economy..GDP.per.Capita.+year+Health..Life.Expectancy., data=dfTrain)

> model
Call:
lm(formula = Happiness.Score ~ Economy..GDP.per.Capita. + year + 
    Health..Life.Expectancy., data = dfTrain)

Coefficients:
             (Intercept)  Economy..GDP.per.Capita.                      year  
                   3.036                     1.569                        NA  
Health..Life.Expectancy.  
                   1.559

variable year has the same value for all records. After removing year variable

model <- lm(Happiness.Score ~ Economy..GDP.per.Capita.+Health..Life.Expectancy., data=dfTrain)

preds <- predict.lm(model, dfTest[, c(1:nrow(dfTest)-1]))

This gives no warning message

Malkamalkah