Model-based boosting in R: a hands-on tutorial using the R package mboost

Abstract

We provide a detailed hands-on tutorial for the R add-on package mboost. The package implements boosting for optimizing general risk functions utilizing component-wise (penalized) least squares estimates as base-learners for fitting various kinds of generalized linear and generalized additive models to potentially high-dimensional data. We give a theoretical background and demonstrate how mboost can be used to fit interpretable models of different complexity. As an example we use mboost to predict the body fat based on anthropometric measurements throughout the tutorial.

Appendix: Building your own family

Via the constructor function Family(), in mboost there exists an easy way for the user to set up new families. The main required arguments are the loss to be minimized and the negative gradient (ngradient) of the loss. The risk is then commonly defined as the sum of the loss over all observations.

We will demonstrate the usage of this function by (re-) implementing the family to fit quantile regression (the pre-defined family is QuantReg()). In contrast to standard regression analysis, quantile regression (Koenker 2005) does not estimate the conditional expectation of the conditional distribution but the conditional quantiles. Estimation is carried out by minimizing the check function $\rho _{\tau }(\cdot )$:

$$\begin{aligned} \rho _{\tau }(y_i - f_{\tau i} ) = \left\{ \begin{array}{ll} (y_i - f_{\tau i} ) \cdot \tau&\quad (y_i - f_{\tau i} ) \ge 0 \\ (y_i - f_{\tau i} ) \cdot (\tau -1)&\quad (y_i - f_{\tau i} ) <0, \end{array} \right. \end{aligned}$$

which is depicted in Fig. 10b. The loss for our new family is therefore given as:

The check-function is not differentiable at the point 0. However in practice, as the response is continuous, we can ignore this by defining:

$$\begin{aligned} - \frac{\partial \rho _{\tau }(y_i, f_{\tau i})}{\partial f} = \left\{ \begin{array}{l@{\quad }l} \tau&(y_i - f_{\tau i}) \ge 0 \\ \tau -1&(y_i - f_{\tau i}) <0. \end{array} \right. \end{aligned}$$

The negative gradient of our loss is therefore:¹⁵

Of further interest is also the starting value for the algorithm, which is specified via the offset argument. For quantile regression it was demonstrated that the offset may be set to the median of the response (Fenske et al. 2011). With this information, we can already specify our new family for quantile regression:

Case study (ctd.): prediction of body fat

To try our new family we go back to the case study regarding the prediction of body fat. First, we reproduce the model for the median, computed with the pre-defined QuantReg() family (see Sect. 3.4.1), to show that our new family delivers the same results:

To get a better idea of the shape of the conditional distribution we model the median, and the 0.05 and 0.95 quantiles in a small, illustrative example containing only the predictor hipcirc:

Note that for different quantiles, fitting has to be carried out separately, as $\tau$ enters directly in the loss. It is also important that fitting quantile regression generally requires higher stopping iterations than standard regression with the $L_2$ loss, as the negative gradients which are fitted to the base-learners are vectors containing only small values, i.e., $\tau$ and $1-\tau$.

The resulting plot (see Fig. 12) shows how quantile regression can be used to get a better impression of the whole conditional distribution function in a regression setting. In this case, the upper and lower quantiles are not just parallel lines to the median regression line but adapt nicely to the slight heteroscedasticity found in this data example: For smaller values of hipcirc the range between the quantiles is smaller than for higher values. Note that the outer quantile-lines can be interpreted as prediction intervals for new observations (Meinshausen 2006; Mayr et al. 2012c). For more on quantile regression in the context of boosting we refer to Fenske et al. (2011).

Fig. 12

Resulting quantile regression lines, for the median (solid line) and the 0.95 and 0.05 quantiles (upper and lower dashed lines)