3-class AUC calculation in R (pROC package)

Question

I met a problem of 3-class ROC analysis in R and obtained a very annoying result (see here). Now I try to use a different way to solve it. The data is iris and the classifier is multinomial logistic regression which is in nnet package. The code is below:

# iris data (3-class ROC)
library(nnet)
library(pROC) # should be installed first: install.packages('pROC')
data(iris)
# 3-class logistic regression
model = multinom(Species~., data = iris, trace = F)
# confusion matrix (z1) & accuracy (E1)
z1 = table(iris[, 5], predict(model, data = iris))
E1 = sum(diag(z1)) / sum(z1)
z1;E1
#             setosa versicolor virginica
#  setosa         50          0         0
#  versicolor      0         49         1
#  virginica       0          1        49
#[1] 0.9866667

# prediction model (still training data set)
pre = predict(model, data = iris, type='probs')
# AUC measure
modelroc = mean(
    c(as.numeric(multiclass.roc(iris$Species, pre[,1])$auc),
        as.numeric(multiclass.roc(iris$Species, pre[,2])$auc),
        as.numeric(multiclass.roc(iris$Species, pre[,3])$auc)
    )
)
modelroc
## RESULT ##
# [1] 0.9803556

My question is:
Is this a right way of using pROC package?
Thanks a lot!!!

Some related reference:
pROC package: http://www.inside-r.org/packages/cran/pROC/docs/multiclass.roc
Hand & Till(2001) original paper: http://link.springer.com/article/10.1023%2FA%3A1010920819831

Alyose

Answer 1

You are taking the mean of three multiclass AUC, which are themselves the mean of the AUC of three ROC curves. So that's kind of the mean of 9 AUC. This might or might not be the right answer to your problem, depending on the actual question you have, but as you never asked it, it is quite difficult to answer. All I can say is, it's not what is described in the Hand & Till(2001) paper you mention.

Attitudinize

Answer 2

I understand this question is relatively old, but I have a solution:

# Starting validation code
library(HandTill2001)

auc(multcap(
  response = iris$Species,
  predicted = as.matrix(pre)
))

This gives me an AUC of 0.9990667, unfortunately, since I am not using the pROC package, plotting is not really supported. But I am also not sure how the package pROC should be used.

Donnydonnybrook