Gradient boosting on Vowpal Wabbit

Question

Is there a way to use gradient boosting on regression using Vowpal Wabbit? I use various techniques that come with Vowpal Wabbit that are helpful. I want to try gradient boosting along with that, but I can't find a way to implement gradient boosting on VW.

Holocaust

Answer 1

The idea of gradient boosting is that an ensemble model is built from black-box weak models. You can surely use VW as the black box, but note that VW does not offer decision trees, which are the most popular choice for the black-box weak models in boosting. Boosting in general decreases bias (and increases variance), so you should make sure that the VW models have low variance (no overfitting). See bias-variance tradeoff.

There are some reductions related to boosting and bagging in VW:

• --autolink N adds a link function with polynomial N, which can be considered a simple way of boosting.
• --log_multi K is an online boosting algorithm for K-class classification. See the paper. You can use it even for binary classification (K=2), but not for regression.
• --bootstrap M M-way bootstrap by online importance resampling. Use --bs_type=vote for classification and --bs_type=mean for regression. Note that this is bagging, not boosting.
• --boosting N (added on 2015-06-17) online boosting with N weak learners, see a theoretic paper

Salonika

Thanks for the help, appreciate it! – Holocaust

A very relevant paper on "Online Gradient Boosting" arxiv.org/abs/1506.04820 was published yesterday. VW is mentioned there. – Salonika

Does that mean one cannot use VW for gradient boosting for regression tasks? – Arbalest