default lambda sequence in glmnet for cross-validation

Question

Does anybody know how cv.glmnet (in R's glmnet) or LassoCV (scikit-learn) chooses a sequence of regularization constants (lambdas), which they use in cross-validation? Thank you very much!

Micrococcus

(In R glmnet at least) Never use the default lambda sequence, this is known to be dangerous. Always supply your own sequence. – Lancey

@Lancey Can you explain in what way / why the default lambda sequence in glmnet is dangerous? Thanks. – Isador

@visitor: "not necessarily guaranteed to find lambda which minimizes CVE". It might, it might not. Plot the deviance/log(lambda) curve to see if you found it. The advice I was always given was define your own lambda sequence, and tweak it if it's not good. – Lancey

Answer 1

According to Friedman, Hastie & Tibshirani (2010) 'strategy is to select a minimum value lambda_min = epsilon * lambda_max, and construct a sequence of K values of lambda decreasing from lambda_max to lambda_min on the log scale. Typical values are epsilon = 0.001 and K = 100.'

The following example generates data, calculates the lambda path and compares it to that of glmnet:

## Load library and generate some data to illustrate:
library("glmnet")
set.seed(1)
n <- 100
x <- matrix(rnorm(n*20), n, 20)
y <- rnorm(n)

## Standardize variables: (need to use n instead of (n-1) as denominator)
mysd <- function(z) sqrt(sum((z-mean(z))^2)/length(z))
sx <- scale(x, scale = apply(x, 2, mysd))
sx <- as.matrix(sx, ncol = 20, nrow = 100)

## Calculate lambda path (first get lambda_max):
lambda_max <- max(abs(colSums(sx*y)))/n
epsilon <- .0001
K <- 100
lambdapath <- round(exp(seq(log(lambda_max), log(lambda_max*epsilon), 
                            length.out = K)), digits = 10)
lambdapath

## Compare with glmnet's lambda path:
fitGLM <- glmnet(sx, y)
fitGLM$lambda

Note that glmnet does not compute solutions for all 100 (default) lambda values though, it stops earlier. Not sure what the rules for stopping are.

See also How does glmnet compute the maximal lambda value

---

Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of statistical software, 33(1), 1.

Jacquie

It seems you normalize y vector and then compute lambda_max. According to package, y is only centered not scaled??? – Lithea

Indeed. But if you replace the line sy <- as.vector(scale(y, scale = mysd(y))) by sy <- y, the calculated lambdapath and fitGLM$lambda are still equal. – Jacquie

Omitted normalization of y now. – Jacquie