calculate accuracy and precision of confusion matrix

Question

Is there any tool / R package available to calculate accuracy and precision of a confusion matrix?

The formula and data structure are here.

Maretz

possible duplicate #6620353 – Nipping

That thread talks about creating confusion matrix. My question is to calculate accuracy and precision on top of a confusion matrix. – Maretz

I found a R package which helps to do this. cran.r-project.org/web/packages/caret/caret.pdf – Maretz

Answer 1

yes, you can calculate Accuracy and precision in R with confusion matrix. It uses Caret package.

Here is the example :

lvs <- c("normal", "abnormal")
truth <- factor(rep(lvs, times = c(86, 258)),
                levels = rev(lvs))
pred <- factor(
               c(
                 rep(lvs, times = c(54, 32)),
                 rep(lvs, times = c(27, 231))),               
               levels = rev(lvs))

xtab <- table(pred, truth)
# load Caret package for computing Confusion matrix
library(caret) 
confusionMatrix(xtab)

And Confusion Matrix for xtab would be like this :

Confusion Matrix and Statistics

          truth
pred       abnormal normal
  abnormal      231     32
  normal         27     54

               Accuracy : 0.8285
                 95% CI : (0.7844, 0.8668)
    No Information Rate : 0.75
    P-Value [Acc > NIR] : 0.0003097

                  Kappa : 0.5336
 Mcnemar's Test P-Value : 0.6025370

            Sensitivity : 0.8953
            Specificity : 0.6279
         Pos Pred Value : 0.8783
         Neg Pred Value : 0.6667
             Prevalence : 0.7500
         Detection Rate : 0.6715
   Detection Prevalence : 0.7645

       'Positive' Class : abnormal

So here is everything, that you want.

Cyathus

how to programatically find precision and recall after results of confusionMatrix(xtab) are got? – Emera

Thanks Nishu. Just one extra information I want to add. confusionMatrix(xtab) has dependency on "e1071" package, so installation of this package may be required. – Pigweed

@Rupesh: yeah, that's required. – Cyathus

Answer 2

@Harsh Trivedi

byClass allows you to pull out the precision and recall from the summary. PPV is precision. Sensitivity is recall. https://en.wikipedia.org/wiki/Precision_and_recall

library(caret)

result <- confusionMatrix(prediction, truth)
precision <- result$byClass['Pos Pred Value']    
recall <- result$byClass['Sensitivity']

I imagine you want to pull out the precision and recall to calculate the f-measure so here it goes.

f_measure <- 2 * ((precision * recall) / (precision + recall))

I also found this handy online calculator for sanity check. http://www.marcovanetti.com/pages/cfmatrix/?noc=2

-bg

Hoedown

No need to calculate this yourself, just use confusionMatrix(..., mode = "everything"), please see below answer. – Lassie

Answer 3

In case anybody is having the same issue as I did, the method confusionMatrix() in caret does indeed give sensitivity/specificity. However, if it is fed an object of type train it will run a different method, confusionMatrix.train() which does not have this information.

The solution is to feed the data and reference manually from the train object (i.e. $pred$pred$ and $pred$obs respectively) to the confusionMatrix() method.

Tolyl

Answer 4

In case someone else is looking: thanks to BGA's answer above I got clearer on how to read the confusionMatrix() output and realized you can get the F-measure right out of the result$ByClass output as F1.

 result$byClass
         Sensitivity          Specificity       Pos Pred Value       Neg Pred Value 
           0.9337442            0.8130531            0.8776249            0.8952497 
           Precision               Recall                   F1           Prevalence 
           0.8776249            0.9337442            0.9048152            0.5894641 
      Detection Rate Detection Prevalence    Balanced Accuracy 
           0.5504087            0.6271571            0.8733987 

Calculating f_measure below with same formula as in above comment also gives 0.9048152.

You can also get the Accuracy from results$overall

result$overall
      Accuracy          Kappa  AccuracyLower  AccuracyUpper   AccuracyNull AccuracyPValue 
  8.841962e-01   7.573509e-01   8.743763e-01   8.935033e-01   5.894641e-01   0.000000e+00 
 McnemarPValue 
  2.745521e-13

Or use Balanced Accuracy from results

Glaab

Answer 5

Much more convenient than calculating the F1 score yourself, or to use result$byClass, you can just use the mode argument in confusionMatrix to print the F1 score, recall, and precision as well.

The default is confusionMatrix(..., mode = "sens_spec"), which shows the sensitivity and specificity. To show the F1 score, recall, and precision, using mode = "prec_recall". To show all of them, use mode = "everything":

library(caret)

x <- matrix(c(231, 27, 32, 54), nrow = 2)
x
#>      [,1] [,2]
#> [1,]  231   32
#> [2,]   27   54

confusionMatrix(x, mode = "everything")
#> Confusion Matrix and Statistics
#> 
#>     A   B
#> A 231  32
#> B  27  54
#> 
#>                Accuracy : 0.8285
#>                  95% CI : (0.7844, 0.8668)
#>     No Information Rate : 0.75
#>     P-Value [Acc > NIR] : 0.0003097
#> 
#>                   Kappa : 0.5336
#> 
#>  Mcnemar's Test P-Value : 0.6025370
#> 
#>             Sensitivity : 0.8953
#>             Specificity : 0.6279
#>          Pos Pred Value : 0.8783
#>          Neg Pred Value : 0.6667
#>               Precision : 0.8783
#>                  Recall : 0.8953
#>                      F1 : 0.8868
#>              Prevalence : 0.7500
#>          Detection Rate : 0.6715
#>    Detection Prevalence : 0.7645
#>       Balanced Accuracy : 0.7616
#> 
#>        'Positive' Class : A

Lassie