ValueError: n_splits=10 cannot be greater than the number of members in each class

Question

I am trying to run the following code:

from sklearn.model_selection import StratifiedKFold 
X = ["hey", "join now", "hello", "join today", "join us now", "not today", "join this trial", " hey hey", " no", "hola", "bye", "join today", "no","join join"]
y = ["n", "r", "n", "r", "r", "n", "n", "n", "n", "r", "n", "n", "n", "r"]

skf = StratifiedKFold(n_splits=10)

for train, test in skf.split(X,y):  
    print("%s %s" % (train,test))

But I get the following error:

ValueError: n_splits=10 cannot be greater than the number of members in each class.

I have looked here scikit-learn error: The least populated class in y has only 1 member but I'm still not really sure what is wrong with my code.

My lists both have lengths of 14 print(len(X)) print(len(y)).

Part of my confusion is that I am not sure what a members is defined as and what a class is in this context.

Questions: How do I fix the error? What is a member? What is a class? (in this context)

Conklin

Please look at my answer for explanation of error. But for solving this, I need to know why you are using StratifiedKFold, and not any other. What's your goal here? – Mccracken

I have a list of 2163 tweets that I have labeled "n" and "r" (I corrected the typo above...no "y" are included). I wanted to replicate the error with a shorter list of X and y. The idea is to perform 10 fold cross validation on the dataset. That is why I wanted to use n_split = 10. And I use stratification because there are 2000 n and 163 y. I thought keeping these ratio's consistent would be desirable – Conklin

Ah ok. In that case, using StratifiedKFold on your original dataset should work. – Mccracken

Answer 1

Stratification means to keep the ratio of each class in each fold. So if your original dataset has 3 classes in the ratio of 60%, 20% and 20% then stratification will try to keep that ratio in each fold.

In your case,

X = ["hey", "join now", "hello", "join today", "join us now", "not today",
     "join this trial", " hey hey", " no", "hola", "bye", "join today", 
     "no","join join"]
y = ["n", "r", "n", "r", "r", "n", "n", "n", "n", "y", "n", "n", "n", "y"]

You have a total of 14 samples (members) with the distribution:

class    number of members         percentage
 'n'        9                        64
 'r'        3                        22
 'y'        2                        14

So StratifiedKFold will try to keep that ratio in each fold. Now you have specified 10 folds (n_splits). So that means in a single fold, for class 'y' to maintain the ratio, at least 2 / 10 = 0.2 members. But we cannot give less than 1 member (sample) so that's why its throwing an error there.

If instead of n_splits=10, you have set n_splits=2, then it would have worked, because than the number of members for 'y' will be 2 / 2 = 1. For n_splits = 10 to work correctly, you need to have atleast 10 samples for each of your classes.

Mccracken

What command did you use to get the summary numbers? class number of members percentage – Alexandrite

@Alexandrite For this example, I calculated manually. For a larger array, you can use pandas or numpy which have such functionality. – Mccracken

Excellent, yes I was wondering if these modules had a summary function like there is in R – Alexandrite

this answer is the most suitable one indeed! was facing the same kind of problem! – Feat