Plotting a ROC curve from a random forest classification

Question

I'm trying to plot ROC curve of a random forest classification. Plotting works, but I think I'm plotting the wrong data since the resulting plot only has one point (the accuracy).

This is the code I use:

set.seed(55)
data.controls <- cforest_unbiased(ntree=100, mtry=3)
data.rf <- cforest(type ~ ., data = dataset ,controls=data.controls) 
pred <- predict(data.rf, type="response")
preds <- prediction(as.numeric(pred), dataset$type)
    perf <- performance(preds,"tpr","fpr")
    performance(preds,"auc")@y.values
    confusionMatrix(pred, dataset$type)

plot(perf,col='red',lwd=3)
abline(a=0,b=1,lwd=2,lty=2,col="gray")

Mercia

Answer 1

To plot a receiver operating curve you need to hand over continuous output of the classifier, e.g. posterior probabilities. That is, you need to predict (data.rf, newdata, type = "prob").

predicting with type = "response" already gives you the "hardened" factor as output. Thus, your working point is implicitly fixed already. With respect to that, your plot is correct.

---

side note: in bag prediction of random forests will be highly overoptimistic!

Basidiospore