Use Scikit Learn to do linear regression on a time series pandas data frame

Question

I'm trying to do a simple linear regression on a pandas data frame using scikit learn linear regressor. My data is a time series, and the pandas data frame has a datetime index:

                value
2007-01-01    0.771305
2007-02-01    0.256628
2008-01-01    0.670920
2008-02-01    0.098047

Doing something simple as

from sklearn import linear_model

lr = linear_model.LinearRegression()

lr(data.index, data['value'])

didn't work:

float() argument must be a string or a number

So I tried to create a new column with the dates to try to transform it:

data['date'] = data.index
data['date'] = pd.to_datetime(data['date'])
lr(data['date'], data['value'])

but now I get:

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

So the regressor can't handle datetime. I saw a bunch of ways to convert integer data to datetime, but couldn't find a way to convert from datetime to integer, for example.

What is the proper way to do this?

PS: I'm interested in using scikit because I'm planning on doing more stuff with it later, so no statsmodels for now.

Calabria

Convert your dates to time since epoch: #15204123 – Intorsion

Thanks. The solution below is more "pandonic". – Calabria

Same desire here. I'm sad that sklearn can't handle a datetimeindex... I wish the science stack played nicer together. – Exchequer

Answer 1

You probably want something like the number of days since the start to be your predictor here. Assuming everything is sorted:

In [36]: X = (df.index -  df.index[0]).days.reshape(-1, 1)

In [37]: y = df['value'].values

In [38]: linear_model.LinearRegression().fit(X, y)
Out[38]: LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

The exact units you use for the predictor don't really matter, it could be days or months. The coefficients and interpretation will change so that everything works out to the same result. Also, notice that we needed a reshape(-1, 1) so that the X is in the expected format.

Crampton

There is another way to do it, that is independent of the frequency of the data: df.index.factorize()[0].reshape(-1,1) – Calabria

index.reshape is not available now. Any suggestions – Chicago

@MichaelLi use df['value'].values.reshape(..) – Bestialize

@Calabria suggestion is very useful for me to calculate Linear Regression in Xarray, Thanks Ivan :) – Understate