McNemar's test in Python and comparison of classification machine learning models [closed]

Question

Is there a good McNemar's test implemented in Python? I don't see it anywhere in Scipy.stats or Scikit-Learn. I may have overlooked some other good packages. Please recommend.

McNemar's Test is almost THE test for comparing two classification algorithms/models given a holdout test set (not through K-fold or resampling method to mimic a test set). Two common alternatives are: t-test for comparing directly true positive proportions p_A and p_B from two algorithms and models A and B by 1) assuming variances follow binomial distributions or 2) estimating variances using repetitive resampling train and test sets.

The latter two, however, have been shown having high Type-1 errors (claims models are statistically different but in essence they are the same). McNemar's test is still considered the best if comparing two classification algorithms or models. See Dietterich10.

Or as an alternative, how do people statistically compare two classification models in practice if not by McNemar's test?

Styptic

Better try crossvalidated..! – Chrysoberyl

StatsModels has McNemar's test, see docs. – Hilary

@Chrysoberyl Not sure if I understand. Do you mean running cross-validation? Once you get some numbers, say 72.01% and 72.22%, how do you know which one is actually (statistically significantly) better or just pure luck? – Styptic

@UlrichStern Awesome! Missed that package in my survey! – Styptic

i mean post on crossvalidated site of stack because here you are rather irrelevant – Chrysoberyl

Just in case further people looking for this. There is a library that includes this test which includes the McNemar's test from statsmodels.org/devel/generated/…. Check this one as well rasbt.github.io/mlxtend/user_guide/evaluate/mcnemar – Akim