How to succinctly write a formula with many variables from a data frame?

Question

Suppose I have a response variable and a data containing three covariates (as a toy example):

y = c(1,4,6)
d = data.frame(x1 = c(4,-1,3), x2 = c(3,9,8), x3 = c(4,-4,-2))

I want to fit a linear regression to the data:

fit = lm(y ~ d$x1 + d$x2 + d$y2)

Is there a way to write the formula, so that I don't have to write out each individual covariate? For example, something like

fit = lm(y ~ d)

(I want each variable in the data frame to be a covariate.) I'm asking because I actually have 50 variables in my data frame, so I want to avoid writing out x1 + x2 + x3 + etc.

Libyan

possible duplicate of Specifying formula in R with glm without explicit declaration of each covariate – Dasi

Also see how to insert dataframe column name into equation. – Whatley

Answer 1

There is a special identifier that one can use in a formula to mean all the variables, it is the . identifier.

y <- c(1,4,6)
d <- data.frame(y = y, x1 = c(4,-1,3), x2 = c(3,9,8), x3 = c(4,-4,-2))
mod <- lm(y ~ ., data = d)

You can also do things like this, to use all variables but one (in this case x3 is excluded):

mod <- lm(y ~ . - x3, data = d)

Technically, . means all variables not already mentioned in the formula. For example

lm(y ~ x1 * x2 + ., data = d)

where . would only reference x3 as x1 and x2 are already in the formula.

Archaeological

The data frame 'd' has 4 columns (y, x1, x2, and x3). So if the formula is "y ~ .", does the right hand side mean "all the columns" except those listed on the left hand side? – Edington

@Edington Yes, . technically means all variables in data not already in the formula. – Archaeological

@theforestecologist if you mean data is a list from which variables in the formula are looked up from that list, then yes. A data frame, list or environment are acceptable options for the data argument. If that's not what you mean, you'll need to expand a little more. – Archaeological

@Gavin. That is what I meant. Thanks. How would I go about this method using data[[x]] as a listed variable vs. the actual variable name (e.g, 'x3')? For example, how would I make the following work?: lm(d[[1]] ~ d[[3]] + ., data = d) – Korenblat

It works off the names of the list; say you have ll <- list(y = rnorm(10), x = rnorm(10), z = rnorm(10), zz = runif(10)), then the following works: lm(y ~ x + ., data = ll). So there isn't much reason to have your data like this unless it already is a list, but it works. The requirement that the elements of the formula be the same length puts some restrictions on what you have in a list. More complex objects probably need code to extract the elements you want; if d[[1]] were a data frame/matrix you need code to make that work – Archaeological

For more info, see ?formula. – Highflier

Answer 2

A slightly different approach is to create your formula from a string. In the formula help page you will find the following example :

## Create a formula for a model with a large number of variables:
xnam <- paste("x", 1:25, sep="")
fmla <- as.formula(paste("y ~ ", paste(xnam, collapse= "+")))

Then if you look at the generated formula, you will get :

R> fmla
y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 + 
    x12 + x13 + x14 + x15 + x16 + x17 + x18 + x19 + x20 + x21 + 
    x22 + x23 + x24 + x25

Nacred

This works very well for reading these values from a file. Thanks! – Purge

Note that the as.formula part is a mustt – Emaemaciate

Answer 3

Yes of course, just add the response y as first column in the dataframe and call lm() on it:

d2<-data.frame(y,d)
> d2
  y x1 x2 x3
1 1  4  3  4
2 4 -1  9 -4
3 6  3  8 -2
> lm(d2)

Call:
lm(formula = d2)

Coefficients:
(Intercept)           x1           x2           x3  
    -5.6316       0.7895       1.1579           NA  

Also, my information about R points out that assignment with <- is recommended over =.

Venosity

Thanks! Yeah, I know everyone always says to use <-, but nobody ever says why and = is easier to type =). – Libyan

@gratur One reason is that things like foo(bar <- 1:10) work (and bar is created) but foo(bar = 1:10) would either fail because bar is not an argument of foo and won't create bar either. – Archaeological

Why is the coefficient of x3 NA? – Isolde

@Ziyuan: Because we are trying to estimate 4 parameters (including intercept) with just 3 data points. The system is indeterminate and lm solves it by dropping one variable. – Ipomoea

Answer 4

An extension of juba's method is to use reformulate, a function which is explicitly designed for such a task.

## Create a formula for a model with a large number of variables:
xnam <- paste("x", 1:25, sep="")

reformulate(xnam, "y")
y ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 + 
    x12 + x13 + x14 + x15 + x16 + x17 + x18 + x19 + x20 + x21 + 
    x22 + x23 + x24 + x25

For the example in the OP, the easiest solution here would be

# add y variable to data.frame d
d <- cbind(y, d)
reformulate(names(d)[-1], names(d[1]))
y ~ x1 + x2 + x3

or

mod <- lm(reformulate(names(d)[-1], names(d[1])), data=d)

Note that adding the dependent variable to the data.frame in d <- cbind(y, d) is preferred not only because it allows for the use of reformulate, but also because it allows for future use of the lm object in functions like predict.

Whatley

Answer 5

I build this solution, reformulate does not take care if variable names have white spaces.

add_backticks = function(x) {
    paste0("`", x, "`")
}

x_lm_formula = function(x) {
    paste(add_backticks(x), collapse = " + ")
}

build_lm_formula = function(x, y){
    if (length(y)>1){
        stop("y needs to be just one variable")
    }
    as.formula(        
        paste0("`",y,"`", " ~ ", x_lm_formula(x))
    )
}

# Example
df <- data.frame(
    y = c(1,4,6), 
    x1 = c(4,-1,3), 
    x2 = c(3,9,8), 
    x3 = c(4,-4,-2)
    )

# Model Specification
columns = colnames(df)
y_cols = columns[1]
x_cols = columns[2:length(columns)]
formula = build_lm_formula(x_cols, y_cols)
formula
# output
# "`y` ~ `x1` + `x2` + `x3`"

# Run Model
lm(formula = formula, data = df)
# output
Call:
    lm(formula = formula, data = df)

Coefficients:
    (Intercept)           x1           x2           x3  
        -5.6316       0.7895       1.1579           NA  

```

Headstall

Answer 6

You can check the package leaps and in particular the function regsubsets() functions for model selection. As stated in the documentation:

Model selection by exhaustive search, forward or backward stepwise, or sequential replacement

Wardmote