statsmodels linear regression - patsy formula to include all predictors in model

Question

Say I have a dataframe (let's call it DF) where y is the dependent variable and x1, x2, x3 are my independent variables. In R I can fit a linear model using the following code, and the . will include all of my independent variables in the model:

# R code for fitting linear model
result = lm(y ~ ., data=DF)

I can't figure out how to do this with statsmodels using patsy formulas without explicitly adding all of my independent variables to the formula. Does patsy have an equivalent to R's .? I haven't had any luck finding it in the patsy documentation.

Gunnery

Answer 1

No this doesn't exist in patsy yet, unfortunately. See this issue.

Mcgann

Answer 2

I haven't found . equivalent in patsy documentation either. But what it lacks in conciseness, it can make-up for by giving strong string manipulation in Python. So, you can get formula involving all variable columns in DF using

all_columns = "+".join(DF.columns - ["y"])

This gives x1+x2+x3 in your case. Finally, you can create a string formula using y and pass it to any fitting procedure

my_formula = "y~" + all_columns
result = lm(formula=my_formula, data=DF)

Gaslight

I'm currently using this approach - it is certainly nice to be able to do string manipulation in Python! I just wanted to make sure I wasn't overlooking something in patsy. – Gunnery

Yeah, and I guessed . is not available precisely because of string manipulation. Another example of dynamically constructing formula: pick all variables staring with x and not starting with z. Messy to do in R, but simple in Python. But again, if anyone knows patsy better, I'd love to find alternatives. – Gaslight

@SudeepJuvekar . I don't think so for the missing dot. R also have a strong and easy string manipulation(using paste,paste0) and provide a smart formula notation. So it is just a missing feature in pasty or not yet implemented. – Galata

The only reason it's not implemented in patsy is that neither I nor anyone else has found the time to do it yet :-) There was one partial attempt here, from which the discussion is probably useful if someone else wants to have a go: github.com/pydata/patsy/pull/28 – Agrobiology

Thats all peachy, but what do you do when you get a Maximum Recursion Depth Error for your constructed formula? I have a sufficiently larger of dimensions than samples (p>>N). Also when I try lm(formula="Y~.", data=df) the error is that the dot isn't understood correctly. Any inspiration on those? – Montagna

DF.columns - ["y"] gave me an error. The syntax which worked for me was DF.columns.difference(["y"]) – Eula

@JeanPaul's reply worked for me instead of DF.columns - ["y"] – Holliehollifield

Answer 3

As this is still not included in patsy, I wrote a small function that I call when I need to run statsmodels models with all columns (optionally with exceptions)

def ols_formula(df, dependent_var, *excluded_cols):
    '''
    Generates the R style formula for statsmodels (patsy) given
    the dataframe, dependent variable and optional excluded columns
    as strings
    '''
    df_columns = list(df.columns.values)
    df_columns.remove(dependent_var)
    for col in excluded_cols:
        df_columns.remove(col)
    return dependent_var + ' ~ ' + ' + '.join(df_columns)

For example, for a dataframe called df with columns y, x1, x2, x3, running ols_formula(df, 'y', 'x3') returns 'y ~ x1 + x2'

Thoer

Could you please provide an example with several columns to exclude? – Odontoid