Tree-based boosting with functional data

Abstract

In this article we propose a boosting algorithm for regression with functional explanatory variables and scalar responses. The algorithm uses decision trees constructed with multiple projections as the “base-learners”, which we call “functional multi-index trees”. We establish identifiability conditions for these trees and introduce two algorithms to compute them. We use numerical experiments to investigate the performance of our method and compare it with several linear and nonlinear regression estimators, including recently proposed nonparametric and semiparametric functional additive estimators. Simulation studies show that the proposed method is consistently among the top performers, whereas the performance of existing alternatives can vary substantially across different settings. In a real example, we apply our method to predict electricity demand using price curves and show that our estimator provides better predictions compared to its competitors, especially when one adjusts for seasonality.

Appendices

Appendix A

Proof of Theorem 1 in Sect. 2.1.1.

Proof

It is clear that if $\{\beta _1,..., \beta _K \} = \left\{ (-1)^{l_1}\eta _1,..., (-1)^{l_K}\eta _K \right\}$ for some $l_1,...,l_K \in \{0,1\}$, then $g = {\tilde{g}}$. Therefore, it suffices to show that $\{\beta _1,..., \beta _K \} = \left\{ (-1)^{l_1}\eta _1,..., (-1)^{l_K}\eta _K \right\}$ for some $l_1,...,l_K \in \{0,1\}$. We prove that if there do not exist $l_1,...,l_K$ for the two sets to be equal, there exists a set of indices for which (12) is a constant function and thus contradicts Condition 2.

For simplicity, we let ${\tilde{\eta }}_j = (-1)^{l_j} \eta _j$. If for any $l_1,...,l_K$, $\{\beta _1,..., \beta _K \} \ne \left\{ {\tilde{\eta }}_1,..., {\tilde{\eta }}_K \right\}$, we match two sets so that the same vectors $\beta _j$ and ${\tilde{\eta }}_j$ align with each other. We let $S = \{\beta _1,...,\beta _K \} \cap \{{\tilde{\eta }}_1,...,{\tilde{\eta }}_K \}$, $\beta _j = {\tilde{\eta }}_j$, for $j = 1,..., |S|$ and $\beta _j \notin \{{\tilde{\eta }}_1,...,{\tilde{\eta }}_K \}$, for $j = |S|+1,..., K$, and $|S| < K$. By Condition 2, there exist a $x_0$ for $J = {|S|+1,...,K}$, (12) is not a constant function.

By (13), Conditions 1 and 2, for any $t_1,..., t_K \in (-\delta , \delta )$

$$\begin{aligned} h\left( \langle x_0 ,\beta _1\rangle + t_1, ..., \langle x_0, \beta _K \rangle + t_K\right)&= h\left( \langle x_0 + t_1\beta _1, \beta _1 \rangle , ..., \langle x_0 + t_K \beta _K\rangle , \beta _K \right) \\&={\tilde{h}} \left( \langle x_0 + t_1 \beta _1, \eta _1 \rangle , ..., \langle x_0 + t_K \beta _K, \eta _K \rangle \right) \\&= {\tilde{h}} \left( \langle x_0, \eta _1 \rangle + t_1 \langle \beta _1, \eta _1\rangle , ..., \langle x_0, \eta _K \rangle + t_K \langle \beta _K,\eta _K\rangle \right) \end{aligned}$$

and similarly

$$\begin{aligned} {\tilde{h}}\left( \langle x_0, \eta _1 \rangle + t_1, ..., \langle x_0, \eta _K \rangle + t_K\right)&= {\tilde{h}}\left( \langle x_0 + t_1 \eta _1 , \eta _1 \rangle , ..., \langle x_0 + t_K \eta _K, \eta _K\rangle \right) \\&=h \left( \langle x_0 + t_1 \eta _1, \beta _1 \rangle , ..., \langle x_0 + t_K \eta _K, \beta _K \rangle \right) \\&= h \left( \langle x_0, \beta _1 \rangle + t_1 \langle \beta _1, \eta _1 \rangle , ..., \langle x_0, \beta _K \rangle + t_K \langle \beta _K, \eta _K\rangle \right) \end{aligned}$$

By Cauchy-Schwarz inequality and Condition 1, $(\langle \beta _j, \eta _j \rangle )^2 = 1$ for $j = 1,..., |S|$ and $(\langle \beta _j, \eta _j \rangle )^2 < 1$ for $j = |S+1|,..., K$. For any $t_1,..., t_K \in (-\delta ,\delta )$,

$$\begin{aligned} h\left( \langle x_0, \beta _1 \rangle + t_1, ..., \langle x_0 , \beta _K \rangle + t_K\right)&= {\tilde{h}} ( \langle x_0,\eta _1 \rangle + t_1 \langle \beta _1, \eta _1 \rangle , ..., \nonumber \\&\quad \langle x_0, \eta _K \rangle + t_K \langle \beta _K, \eta _K\rangle ) \nonumber \\&= h ( \langle x_0, \beta _1 \rangle + t_1 \langle \beta _1, \eta _1 \rangle ^2, ..., \nonumber \\&\quad \langle x_0, \beta _K\rangle + t_K \langle \beta _K, \eta _K\rangle ^2 ) \nonumber \\&\vdots \nonumber \\&= h ( \langle x_0, \beta _1 \rangle + t_1 \langle \beta _1, \eta _1 \rangle ^{2n}, ..., \nonumber \\&\quad \langle x_0, \beta _K\rangle + t_K \langle \beta _K, \eta _K \rangle ^{2n} )\nonumber \\&\vdots \nonumber \\&= h \left( \langle x_0, \beta _1 \rangle + t_1I_1, ..., \langle x_0, \beta _K\rangle + t_KI_K\right) \end{aligned}$$

(22)

where $I_j = 1$ for $j = 1,...,|S|$ and $I_j = 0$ for $j = |S+1|,..., |K|$.

Let $x = x_0 + t e$ for any unit function $e \in L^2({\mathcal {I}})$, $\Vert e \Vert = 1$ and $t \in (-\delta , \delta )$. Then x fills the space of $B(x_0, \delta )$. For $j = 1,...,K$, we define

$$\begin{aligned} L_j(x)&= (1-I_j)x + I_j x_0 \\&= (1-I_j) (x_0 + te) + I_j x_0 \\&= x_0 + (1 - I_j) te \\ h( \langle L_1(x),\beta _1 \rangle ,..., \langle L_K(x), \beta _K \rangle )&= h( \langle x_0 + (1 - I_1) te, \beta _1 \rangle ,...,\langle x_0 + (1 - I_K) te, \beta _K \rangle ) \\&= h( \langle x_0, \beta _1 \rangle + \langle (1 - I_1) te, \beta _1 \rangle ,..., \langle x_0, \beta _K \rangle + \\&\langle (1 - I_K) te, \beta _K \rangle ) , \text {by} \ (22) \\&= h( \langle x_0, \beta _1 \rangle + \langle I_1 (1 - I_1) te, \beta _1 \rangle ,..., \langle x_0, \beta _K \rangle + \\ {}&\langle I_K(1 - I_K) te, \beta _K \rangle ) \\&=h( \langle x_0, \beta _1 \rangle ,..., \langle x_0, \beta _K \rangle ), \end{aligned}$$

which is a constant function of x and that contradicts Condition 2. $\square$

Appendix B

The summary statistics of test MSEs from 100 independent runs of the simulation are provided in Tables 2, 5, 8 and 11, with bold font indicating the lowest two average test errors in each setting. Summary statistics of the tree depths selected by TFBoost are provided in Tables 3,6, 9 and 12, and for the early stopping times for TFBoost are provided in Tables 4, 7, 10 and 13.

Table 2 Summary statistics of test errors for data generated from $r_1$

Table 3 Summary statistics of the tree depths selected by TFBoost for data generated from $r_1$

Table 4 Summary statistics of the early stoping times $T_{\text {stop}}$ selected by TFBoost methods for data generated from $r_1$

Table 5 Summary statistics of test errors for data generated from $r_2$

Table 6 Summary statistics of the tree depths selected by TFBoost for data generated from $r_2$

Table 7 Summary statistics of the early stoping times $T_{\text {stop}}$ selected by TFBoost methods for data generated from $r_2$

Table 8 Summary statistics of test errors for data generated from $r_3$

Table 9 Summary statistics of the tree depths selected by TFBoost for data generated from $r_3$

Table 10 Summary statistics of the early stoping times $T_{\text {stop}}$ selected by TFBoost methods for data generated from $r_3$

Table 11 Summary statistics of test errors for data generated from $r_4$

Table 12 Summary statistics of the tree depths selected by TFBoost for data generated from $r_4$

Table 13 Summary statistics of the early stoping times $T_{\text {stop}}$ selected by TFBoost methods for data generated from $r_4$

Figures 9, 10, 11 and 12 show the test MSEs averaged over 100 runs of the experiment versus the number of iterations for TFBoost. For the convenience of taking the averages, for each run, we let the test errors for the iterations past the early stopping time to keep the same test error as the one obtained at the early stopping time. It can be observed from figures that the test errors usually drop quickly within 100 iterations.

Fig. 9

Test MSEs averaged from 100 runs of the experiment for TFBoost in $r_1$ settings

Fig. 10

Test MSEs averaged from 100 runs of the experiment for TFBoost in $r_2$ settings

Fig. 11

Test MSEs averaged from 100 runs of the experiment for TFBoost in $r_3$ settings

Fig. 12

Test MSEs averaged from 100 runs of the experiment for TFBoost in $r_4$ settings

Appendix C

We consider another regression function that is linear:

$$\begin{aligned} r_5(X) = \int _{{\mathcal {I}}} \left( \text {sin} \left( \frac{3}{2} \pi t \right) + \text {sin} \left( \frac{1}{2} \pi t \right) \right) X(t)dt. \end{aligned}$$

The other specifications of the model remain the same as described in Sect. 3. Table 14 include the summary statistics of test MSEs from 100 independent runs of the simulation, with bold font indicating the lowest two average test errors in each setting. Tables 15 and 16 include the summary statistics of the tree depths and early stopping times selected by TFBoost methods.

Table 14 Summary statistics of test errors for data generated from $r_5$

Table 15 Summary statistics of the tree depths selected by TFBoost for data generated from $r_5$

Table 16 Summary statistics of the early stoping times $T_{\text {stop}}$ selected by TFBoost methods for data generated from $r_5$

Appendix D

The summary statistics of test MSEs from 100 random partitions of the German electricity data in Sect. 4 are provided in Table 17.

Table 17 Summary statistics of test MSEs displayed in the form of mean (sd).