Building multi-regression model throws error: `Pandas data cast to numpy dtype of object. Check input data with np.asarray(data).`

Question

I have pandas dataframe with some categorical predictors (i.e. variables) as 0 & 1, and some numeric variables. When I fit that to a stasmodel like:

est = sm.OLS(y, X).fit()

It throws:

Pandas data cast to numpy dtype of object. Check input data with np.asarray(data). 

I converted all the dtypes of the DataFrame using df.convert_objects(convert_numeric=True)

After this all dtypes of dataframe variables appear as int32 or int64. But at the end it still shows dtype: object, like this:

4516        int32
4523        int32
4525        int32
4531        int32
4533        int32
4542        int32
4562        int32
sex         int64
race        int64
dispstd     int64
age_days    int64
dtype: object

Here 4516, 4523 are variable labels.

Any idea? I need to build a multi-regression model on more than hundreds of variables. For that I have concatenated 3 pandas DataFrames to come up with final DataFrame to be used in model building.

Alphabetize

The output you're seeing is as expected. The dtype listed at the end of your output is the dtype of the dtypes series (the result of calling pd.DataFrame.dtypes) and has nothing to do with the types inside of your dataframe. Just try pd.DataFrame(range(100)).dtypes – Nissie

check np.asarray(X).dtype which should be float64, or int64 which, I think, will be converted to float64 inside statsmodels. Best to check again est.model.exog.dtype to make sure float64 is used in the calculations. – Airborne

Note, if the 'object' dtypes are categorical variables like strings, then it is better to use the formula interface which will automatically create a (numerical) dummy encoding for string variables. Otherwise, use pandas categorical to convert to dummy endoding. – Airborne

Answer 1

If X is your dataframe, try using the .astype method to convert to float when running the model:

est = sm.OLS(y, X.astype(float)).fit()

Randazzo

so .. converting categorical variables to floats? – Heraclea

all categorical variables should be converted into dummy variables before sticking them in the model, so yes – Randazzo

And integers are not good enough, they must be floats! Int64 produces the same error as object or category ... sigh. – Capernaum

There could be a different possibility that the data types would just need to be corresponding. e.g., strings go with strings, float goes with floats, etc. – Scarcely

This is only true if the categorical variable is binary. If not, other coding schemes like dummy or one-hot coding should be used. Otherwise, regression has no way of knowing that the numerical variable is only on nominal scale and mistakes it as continuous. – Pyrology

Answer 2

if both y(dependent) and X are taken from a data frame then type cast both:-

est = sm.OLS(y.astype(float), X.astype(float)).fit()

Salliesallow

so .. converting categorical variables to floats? – Heraclea

Answer 3

As Mário and Daniel suggested, yes, the issue is due to categorical values not previously converted into dummy variables.

I faced this issue reviewing StatLearning book lab on linear regression for the "Carseats" dataset from statsmodels, where the columns 'ShelveLoc', 'US' and 'Urban' are categorical values, I assume the categorical values causing issues in your dataset are also strings like in this one. Considering the previous, I will use this as an example since you didn't provide dataframes for the question.

The columns we have at the beginning are the following, as stated before 'ShelveLoc', 'US' and 'Urban'are categorical:

Index(['Sales', 'CompPrice', 'Income', 'Advertising', 'Population', 'Price',
       'ShelveLoc', 'Age', 'Education', 'Urban', 'US'],
      dtype='object')

In a simple line for Python, I converted them to categorical values and dropped the ones that had "No" and "Bad" labels (as this is what was being requested from the lab in the book).

carseats = pd.get_dummies(carseats, columns=['ShelveLoc', 'US', 'Urban'], drop_first = True)

This will return a dataframe with the following columns:

Index(['Sales', 'CompPrice', 'Income', 'Advertising', 'Population', 'Price',
       'Age', 'Education', 'ShelveLoc_Good', 'ShelveLoc_Medium', 'US_Yes',
       'Urban_Yes'],
      dtype='object')

And that's it, you have dummy variables ready for OLS. Hope this is useful.

Selfassurance

Answer 4

This is because you have NOT generated the dummy values step to all predictors so how can the regression take place over literals ? that is what the error message is saying it is trying to convert to numpy valid entries.

Just go back to your pipeline and include the dummies properly.

Eleneeleni

Answer 5

Before you run the OLS, check X.info(), this will reveal the available columns and their dataTypes. Confirm all of them are numeric in nature. If they are Object or bool then do appropriate conversions on the values such that they are converted into 1 and 0 and then run the OLS.

Featherston