Why is statsmodels throwing an IndedxError when I try to fit a linear mixed-effect model?

Question

Given the code:

import statsmodels.api as sm
import statsmodels.formula.api as smf

df.reset_index(drop=True, inplace=True)
display(df.describe())
md = smf.mixedlm("c ~ iscorr", df, groups=df.subnum)
mdf = md.fit()

Where df is a pandas.DataFrame, I get the following error out of smf.mixedlm:

IndexError                                Traceback (most recent call last)
<ipython-input-34-5373fe9b774a> in <module>()
      4 df.reset_index(drop=True, inplace=True)
      5 display(df.describe())
----> 6 md = smf.mixedlm("c ~ iscorr", df, groups=df.subnum)
      7 # mdf = md.fit()

/home/lthibault/.pyenv/versions/3.5.0/lib/python3.5/site-packages/statsmodels/regression/mixed_linear_model.py in from_formula(cls, formula, data, re_formula, subset, *args, **kwargs)
    651                                                subset=None,
    652                                                exog_re=exog_re,
--> 653                                                *args, **kwargs)
    654 
    655         # expand re names to account for pairs of RE

/home/lthibault/.pyenv/versions/3.5.0/lib/python3.5/site-packages/statsmodels/base/model.py in from_formula(cls, formula, data, subset, *args, **kwargs)
    148         kwargs.update({'missing_idx': missing_idx,
    149                        'missing': missing})
--> 150         mod = cls(endog, exog, *args, **kwargs)
    151         mod.formula = formula
    152 

/home/lthibault/.pyenv/versions/3.5.0/lib/python3.5/site-packages/statsmodels/regression/mixed_linear_model.py in __init__(self, endog, exog, groups, exog_re, use_sqrt, missing, **kwargs)
    537 
    538         # Split the data by groups
--> 539         self.endog_li = self.group_list(self.endog)
    540         self.exog_li = self.group_list(self.exog)
    541         self.exog_re_li = self.group_list(self.exog_re)

/home/lthibault/.pyenv/versions/3.5.0/lib/python3.5/site-packages/statsmodels/regression/mixed_linear_model.py in group_list(self, array)
    671         if array.ndim == 1:
    672             return [np.array(array[self.row_indices[k]])
--> 673                     for k in self.group_labels]
    674         else:
    675             return [np.array(array[self.row_indices[k], :])

/home/lthibault/.pyenv/versions/3.5.0/lib/python3.5/site-packages/statsmodels/regression/mixed_linear_model.py in <listcomp>(.0)
    671         if array.ndim == 1:
    672             return [np.array(array[self.row_indices[k]])
--> 673                     for k in self.group_labels]
    674         else:
    675             return [np.array(array[self.row_indices[k], :])

IndexError: index 7214 is out of bounds for axis 1 with size 7214

Why is this error occurring? len(df) reports that there are 7296 rows, so there should be no issue indexing the 7214th, and the explicit re-indexing ensures that the indices span from zero to 7295.

You may download df here to fiddle around with it if you'd like.

Dahlia

Thanks for providing the link to the data file. Made it much easier to find the issue. – Ornstead

@Ornstead No problem! Thanks for finding the issue! Made it much easier to get on with my day :) – Dahlia

Answer 1

You have 82 null values in iscorr:

>>> df.iscorr.isnull().sum()
82

Drop them and you will be fine:

df = df[df.iscorr.notnull()]

Per the function's docstring:

    Notes
    ------
    `data` must define __getitem__ with the keys in the formula
    terms args and kwargs are passed on to the model
    instantiation. E.g., a numpy structured or rec array, a
    dictionary, or a pandas DataFrame.

    If `re_formula` is not provided, the default is a random
    intercept for each group.

    This method currently does not correctly handle missing
    values, so missing values should be explicitly dropped from
    the DataFrame before calling this method.
    """

Output:

>>> mdf.params
Intercept         0.032000
iscorr[T.True]    0.030670
Intercept RE     -0.057462

Ornstead

Thanks! Christ, that's a really misleading error... :/ – Dahlia

I don't think iscorr is a function anymore, so instead you can check for columns with missing values by doing df.isnull().any() – Saransk