Multi-target regression via input space expansion: treating targets as inputs
- 1.8k Downloads
- 14 Citations
Abstract
In many practical applications of supervised learning the task involves the prediction of multiple target variables from a common set of input variables. When the prediction targets are binary the task is called multi-label classification, while when the targets are continuous the task is called multi-target regression. In both tasks, target variables often exhibit statistical dependencies and exploiting them in order to improve predictive accuracy is a core challenge. A family of multi-label classification methods address this challenge by building a separate model for each target on an expanded input space where other targets are treated as additional input variables. Despite the success of these methods in the multi-label classification domain, their applicability and effectiveness in multi-target regression has not been studied until now. In this paper, we introduce two new methods for multi-target regression, called stacked single-target and ensemble of regressor chains, by adapting two popular multi-label classification methods of this family. Furthermore, we highlight an inherent problem of these methods—a discrepancy of the values of the additional input variables between training and prediction—and develop extensions that use out-of-sample estimates of the target variables during training in order to tackle this problem. The results of an extensive experimental evaluation carried out on a large and diverse collection of datasets show that, when the discrepancy is appropriately mitigated, the proposed methods attain consistent improvements over the independent regressions baseline. Moreover, two versions of Ensemble of Regression Chains perform significantly better than four state-of-the-art methods including regularization-based multi-task learning methods and a multi-objective random forest approach.
Keywords
Multi-target regression Multi-label classification Stacking Chaining1 Introduction
Multi-target regression (MTR), also known as multivariate or multi-output regression, refers to the task of predicting multiple continuous variables using a common set of input variables. Such problems arise in various fields including ecological modeling (Kocev et al. 2009; Dzeroski et al. 2000) (e.g. predicting the abundance of plant species using water quality measurements), economics (Ghosn and Bengio 1996) (e.g. predicting stock prices from econometric variables) and energy (e.g. predicting energy production in solar/wind farms using historical measurements and weather forecast information). Given the importance and diversity of its applications, it is not surprising that research on this topic has started as early as 40 years ago in Statistics (Izenman 1975).
Recently, a closely related task called multi-label classification (MLC) (Tsoumakas et al. 2010; Zhang and Zhou 2014) has received increased attention by Machine Learning researchers. Similarly to MTR, MLC deals with the prediction of multiple variables using a common set of input variables. However, prediction targets in MLC are binary. In fact, the two tasks can be thought of as instances of the more general learning task of multi-target prediction where targets can be continuous, binary, ordinal, categorical or even of mixed type. The baseline approach of learning a separate model for each target applies to both MTR and MLC. Moreover, they share the same core challenge of exploiting dependencies between targets (in addition to dependencies between targets and inputs) in order to improve prediction accuracy, as acknowledged by researchers working in both tasks (e.g. Izenman 2008; Dembczynski et al. 2012). Despite their commonalities, MTR and MLC have typically been treated in isolation and only few works (Blockeel et al. 1998; Weston et al. 2002; Teh et al. 2005; Balasubramanian and Lebanon 2012) have given a general formulation of their key ideas, recognizing the dual applicability of their approaches.
Motivated by the tight connection between the two tasks, this paper looks at a family of MLC methods that, despite being almost directly applicable to MTR problems, have not been applied so far in this domain. In particular, we consider methods that decompose the MLC task into a series of binary classification tasks, one for each label. This category, includes the typical one-versus-all or Binary Relevance approach that assumes label independence but also approaches that model label dependencies by building models that treat other labels as additional input variables (meta-inputs). In this work we adapt two popular methods of this kind (Godbole and Sarawagi 2004; Read et al. 2011) for MTR, contributing two new MTR methods: Stacked single-target (SST) and Ensemble of Regressor Chains (ERC). Both methods have been very successful in the MLC domain and provided inspiration for many subsequent works (Cheng and Hüllermeier 2009; Dembczynski et al. 2010; Kumar et al. 2012; Read et al. 2014).
Although the adaptation is trivial (as it basically consists of employing a regression instead of a binary classification algorithm to solve each single-target prediction task), it widens the applicability of existing approaches and increases our understanding of challenges shared by both learning tasks, such as the modeling of target dependencies. This kind of abstraction of key ideas from solutions tailored to related problems can sometimes offer additional advantages, such as improving the modularity and conceptual simplicity of learning techniques and avoiding reinvention of the same solutions.^{1}
In addition to evaluating the direct adaptations of the corresponding MLC methods in the MTR domain, we also take a careful look at the treatment of targets as additional input variables and spot a shortcoming that was overlooked in the original MLC formulations of both methods. Specifically, we notice that in both methods the values of the meta-inputs are generated differently between training and prediction, causing a discrepancy that is shown to drastically downgrade their performance. To tackle this problem, we develop extended versions of the two methods that manage to decrease the discrepancy by using out-of-sample estimates of the targets during training. These estimates are obtained via an internal cross-validation methodology.
The performance of the proposed methods is comprehensively analyzed based on a large experimental study that includes 18 diverse real-world datasets, 14 of which are firstly used in this paper and are made publicly available for future benchmarks. The experimental results reveal that, affected by the discrepancy problem, the direct adaptations of the corresponding MLC methods fail to obtain better accuracy than the baseline approach that performs independent regressions. On the other hand, the extended versions obtain consistent improvements against the baseline, confirming the effectiveness of the proposed solution. Furthermore, extended versions of ERC obtain significantly better accuracy than state-of-the-art methods, including a method based on ensembles of multi-objective decision trees (Kocev et al. 2007) and a recent regularization-based multi-task learning method (Jalali et al. 2010, 2013). Moreover, it is shown that, compared to the rest of the methods, the extended versions of ERC are associated with the smallest risk of decreasing the accuracy of the baseline, an appealing property.
The rest of the paper is organized as follows: Sect. 2 presents the SST and ERC methods and describes the discrepancy problem and the proposed solution. Section 3 discusses related work from the MTR field, including well-known statistical procedures and multi-task learning methods, and points out differences with previous work on the discrepancy problem. The details of the experimental setup (method configuration, evaluation methodology, datasets) are given in Sects. 4 and 5 presents and discusses the experimental results. Finally, Sect. 6 offers our conclusion and outlines future work directions.
2 Methods
We first formally describe the MTR task and provide the notation that will be used subsequently for the description of the methods. Let \(\mathbf {X}\) and \(\mathbf {Y}\) be two random vectors where \(\mathbf {X}\) consists of d input variables \(X_1,\ldots ,X_d\) and \(\mathbf {Y}\) consists of m target variables \(Y_1,\ldots ,Y_m\). We assume that samples of the form \((\mathbf {x,y})\) are generated i.i.d. by some source according to a joint probability distribution \(\mathbf {P}(\mathbf {X,Y})\) on \(\mathscr {X} \times \mathscr {Y}\) where \(\mathscr {X}=R^d\)^{2} and \(\mathscr {Y}=R^m\) are the domains of \(\mathbf {X}\) and \(\mathbf {Y}\) and are often referred to as the input and the output space. In a sample \((\mathbf {x,y})\), \(\mathbf {x}=[x_1,\ldots ,x_d]\) is the input vector and \(\mathbf {y}=[y_1,\ldots ,y_m]\) is the output vector which are realizations of \(\mathbf {X}\) and \(\mathbf {Y}\) respectively. Given a set \(D=\{(\mathbf {x}^1,\mathbf {y}^1),\ldots ,(\mathbf {x}^n,\mathbf {y}^n)\}\) of n training examples, the goal in MTR is to learn a model \(\mathbf {h}:\mathscr {X}\rightarrow \mathscr {Y}\) that given an input vector \(\mathbf {x}\), is able to predict an output vector \(\hat{\mathbf {y}} = \mathbf {h(x)}\) that best approximates the true output vector \(\mathbf {y}\).
In the baseline Single-Target (ST) method, a multi-target model \(\mathbf {h}\) is comprised of m single-target models \(h_j:\mathscr {X} \rightarrow R\) where each model \(h_j\) is trained on a transformed training set \(D_j = \{(\mathbf {x}^{1},y_{j}^{1}),\ldots ,(\mathbf {x}^{n},y_{j}^{n})\}\) to predict the value of a single target variable \(Y_j\). This way, target variables are modeled independently and no attempt is made to exploit potential dependencies between them. Despite the simplicity of the ST approach, several empirical studies (e.g. Luaces et al. 2012) have shown that Binary Relevance, its MLC counterpart, often obtains comparable performance with more sophisticated MLC methods that model label dependencies, especially in cases where the underlying single-target prediction model is well fitted to the data (Dembczynski et al. 2012; Read and Hollmén 2014, 2015). A theoretical explanation of these results was offered by Dembczynski et al. (2012) who showed that modeling the marginal conditional distributions \(P(Y_i|\mathbf {x})\) of the labels (as done by Binary Relevance) can be sufficient for getting good results in multi-label losses whose risk minimizers can be expressed in terms of marginal distributions (e.g. Hamming loss).
2.1 Stacked single-target
Stacked single-target (SST) is inspired from the Stacked Binary Relevance method (Godbole and Sarawagi 2004) where the idea of stacked generalization (Wolpert 1992) was applied in a MLC context. The training of SST consists of two stages. In the first stage, m independent single-target models \(h_j:\mathscr {X} \rightarrow R\) are learned as in ST. However, instead of directly using these models for prediction, SST involves an additional training stage where a second set of m meta models \(h'_j: \mathscr {X} \times R^{m} \rightarrow R\) are learned, one for each target \(Y_j\). Each meta model \(h'_j\) is learned on a transformed training set \(D'_j=\{(\mathbf {x}'^{1},y^1_j),\dots ,(\mathbf {x}'^{n},y^n_j)\}\), where the original input vectors of the training examples (\(\mathbf {x}^{i}\)) have been augmented by estimates of the values of their target variables (\(\hat{y}^i_1,\ldots ,\hat{y}^i_m\)) to form expanded input vectors \(\mathbf {x}'^{i}=[\mathbf {x}^{i},\hat{y}^i_1,\ldots ,\hat{y}^i_m]\). These estimates are obtained by applying the first stage models to the examples of the training set.
2.2 Ensemble of regressor chains
Regressor Chains (RC) is derived from Classifier Chains (Read et al. 2011), a recently proposed MLC method based on the idea of chaining binary models. The training of RC consists of selecting a random chain (permutation) of the set of target variables and then building a separate regression model for each target. Assuming that the chain \(C = \{Y_1 , Y_2 ,\ldots , Y_m\}\) (C represents an ordered set) is selected, the first model concerns the prediction of \(Y_1\), has the form \(h_1: \mathscr {X} \rightarrow R\) and is the same as the model built by the ST method for this target. The difference in RC is that subsequent models \(h_{j}, j>1\) are learned on transformed training sets \(D'_j = \{(\mathbf {x}'^{1}_j,y^{1}_{j}),\ldots ,(\mathbf {x}'^{n}_j,y^{n}_{j})\}\), where the original input vectors of the training examples have been augmented by the actual values of all previous targets of the chain to form expanded input vectors \(\mathbf {x}'^{i}_j = [x^i_1,\ldots ,x^i_d,y^{i}_{1},\ldots ,y^{i}_{j-1}]\). Thus, the models built for targets \(Y_{j}\) have the form \(h_j: \mathscr {X} \times R^{j-1} \rightarrow R\).
One notable property of RC is that it is sensitive in the selected chain ordering. To alleviate this issue, Read et al. (2011) proposed an ensemble scheme called Ensemble of Classifier Chains where a set of k Classifier Chains models with different random chains are built on bootstrap samples of the training set and the final predictions come from majority voting. This scheme has been shown to consistently improve the accuracy of a single Classifier Chain in the classification domain. We apply the same idea on RC and compute the final predictions by taking the mean of the k estimates for each target. The resulting method is called Ensemble of Regressor Chains (ERC).
2.3 Theoretical insights into stacking and chaining
unconditional, where \(P(\mathbf {Y}) \ne \prod _{i=1}^{m} P(Y_i)\); and
conditional, where \(P(\mathbf {Y}|\mathbf {x}) \ne \prod _{i=1}^{m} P(Y_i|\mathbf {x})\),
Another interesting interpretation is offered by Read and Hollmén (2015) who show that Binary Relevance can (under certain conditions) achieve optimal performance in any dataset, and that improvements over the independent approach are often the result of using an inadequate base learner. Under this view, stacking and chaining can be considered as ‘deep’ independent learners who owe their improved performance over Binary Relevance (when the same base learner is used) to the use of labels as nodes in the inner layers of a deep neural network. These nodes represent readily available^{4} (in the training phase), high-level transformations of the original inputs. This interpretation of stacking and chaining applies directly to the MTR versions of these methods that we present here.
From a bias-variance perspective, we observe that by introducing additional features to single-target models, SST and ERC have the effect of decreasing their bias at the expense of an increased variance. This suggests that whenever the increase in variance is outweighed by the decrease in bias, one should expect gains in generalization performance over ST. This also hints that both methods will probably benefit from being combined with a base regressor that includes a variance reduction mechanism like bagged (Breiman 1996) regression trees.^{5} As shown in Munson and Caruana (2009), bagged trees not only ignore irrelevant features but can also exploit features that contain useful but noisy information. Both properties are very important in the context of SST and ERC because some of the extra features that they introduce might be irrelevant (e.g. whenever two target variables are statistically independent) and/or noisy (as discussed in the following subsection).
2.4 Generation of meta-inputs
Both SST and ERC are based on the same core idea of treating other prediction targets as additional input variables. These meta-inputs differ from ordinary inputs in the sense that while their actual values are available at training time, they are missing during prediction. Thus, during prediction both methods have to rely on estimates of these values which come either from ST (in the case of SST) or from RC (in the case of ERC) models built on the training set. An important question that is answered differently by each method is the following: What type of values should be used at training time for the meta-inputs? SST uses estimates of the variables obtained by applying the first stage models on the training examples, while ERC uses their actual values. We observe that in both cases a core assumption of supervised learning is violated: that the training and testing data should be identically and independently distributed. In the SST case, the in-sample estimates that are used to form the training examples of the second stage models will typically be more accurate than the out-of-sample estimates used at prediction time. The situation is even more problematic in the case of ERC since the actual target values are used during training. In both cases, some of the input variables that are used by the underlying regression algorithm during model induction, become noisy (or noisier in the case of SST) at prediction time and, as a result, the induced model might wrongly estimate (overestimate) their usefulness.
To mitigate this problem, we propose the use of out-of-sample estimates of the targets during training in order to increase the compatibility between the training values of the target variables and the values used during prediction. One way to obtain such estimates is to use a subset of the training set for building the first stage ST models (in the case of SST) or the RC models (in the case of ERC) and apply them to the held-out part. However, this approach would lead to reduced second stage training sets for SST as only the examples of the held-out set would be available for training the second stage models. The same holds for ERC where the chained RC models would be trained on training sets of decreasing size. The solution that we propose to this problem is the use of an internal f-fold cross-validation approach that allows obtaining out-of-sample estimates of the target variables for all the training examples. Compared to the actual target values or the in-sample estimates of the targets, the cross-validation estimates are expected to better resemble the values that are used during prediction. As a result, we expect that the contribution of the meta-inputs to the prediction of each target will be better estimated by the underlying regression algorithm.
2.5 Discussion
Besides the type of values that each method uses for the meta-inputs at training time, SST and ERC have additional conceptual differences. A notable one is that the model built for each target \(Y_j\) by SST, uses all other targets as inputs while in RC each model involves only targets that precede \(Y_j\) in a random chain. As a result, the model built for \(Y_j\) by RC, cannot benefit from statistical relationships with targets that appear later than \(Y_j\) in the chain. This potential disadvantage of RC is partially overcome by ERC since each target is included in multiple random chains and, therefore, the probability that other targets will precede it is increased. At a first glance, SST seems to represent a more straightforward way of including all the available information about other targets. However, we should take into account that, since both methods rely on estimates of the meta-inputs at prediction time (as discussed in previous subsection), the more the meta-inputs that are included in the input space, the higher the amount of error accumulation that is risked at prediction time. From this perspective, ERC seems to adopt a more cautious approach than SST. On the other hand, the estimates of the meta-inputs that are used by the second stage models in SST come from independent models, while the estimates of the meta-inputs used by each model in RC (and ERC) come from models that include information about other targets and thus involve a higher risk of becoming noisy. Overall, there seems to be a trade-off between using the additional information available in the targets and the noise that this information comes with. Which of the two methods (and which variant) achieves a better balance in this trade-off is revealed by the experimental analysis in Sect. 5.
2.6 Complexity analysis
In this section we discuss the time complexity of all variants of the proposed methods at training and test time, given a single-target regression algorithm with training complexity \(O(g_{tr}(n,d))\) and test complexity \(O(g_{te}(n,d))\) for a dataset with n examples and d input variables. The training and test complexities of the ST method are \(O(m {\cdot } g_{tr}(n,d))\) and \(O(m {\cdot } g_{te}(n,d))\) respectively, as it involves training and querying m independent single-target models.
With respect to SST, the method builds \(2 {\cdot } m\) models at training time, all of which are queried at prediction time. In all variants of the method, half of the models are built on the original input space and half of the models are built on an input space augmented by m meta-inputs. Thus, in the case of SST\(_{true}\), where the meta-inputs are readily available, the training and test complexities are \(O(m {\cdot } (g_{tr}(n,d) {+} g_{tr}(n,d{+}m)))\) and \(O(m {\cdot } (g_{te}(n,d) {+} g_{te}(n,d{+}m)))\) respectively. Given that in most cases (see Table 3) the number of targets is much smaller than the number of inputs, i.e. \(m \ll d\), the effective training and test complexities of SST\(_{true}\) become \(O(m {\cdot } g_{tr}(n,d))\) and \(O(m {\cdot } g_{te}(n,d))\) respectively, thus same with ST’s complexities. SST\(_{train}\) and SST\(_{cv}\) have the same test complexity with SST\(_{true}\) but a larger training complexity because of the process of generating estimates for the meta-inputs. In the SST\(_{train}\) case, the training complexity is \(O(m {\cdot } g_{tr}(n,d) {+} m {\cdot } g_{te}(n,d))\) because the m first-stage models are applied to obtain estimates for all the training examples. For most regression algorithms (e.g. regression trees), the computational cost of making predictions for n instances is much smaller than the cost of training on n examples. For instance, the training complexity of a typical binary regression tree learner is \(O(n {\cdot } d^2)\) (Su and Zhang 2006) while the test complexity is \(O(n {\cdot } \log _2 d)\). Thus, practically, the training complexity of SST\(_{train}\) is similar to that of SST\(_{true}\). When it comes to SST\(_{cv}\), in addition to the m first-stage models, f additional models are built on \(\frac{f-1}{f} {\cdot } n\) examples each. Therefore, the training complexity of SST\(_{cv}\) is \(O(m {\cdot } g_{tr}(n,d) + m {\cdot } f {\cdot } g_{tr}(\frac{f-1}{f} {\cdot } n,d) + m {\cdot } g_{te}(n,d)) \approx O(f {\cdot } m {\cdot } g_{tr}(n,d) {+} m {\cdot } g_{te}(n,d))\). Given
that \(g_{te}(n,d)\)\(\ll \)\(g_{tr}(n,d)\), we conclude that the training complexity of SST\(_{cv}\) is roughly f times ST’s training complexity. Also, note that SST\(_{train}\) and SST\(_{cv}\) can be parellelized stage-wise both at training and at prediction time, i.e. all single-target models within the same level can be trained and queried independently, while SST\(_{true}\) is fully parallelizable at training time (all single-target models can be trained independently) and stage-wise parallelizable at test time.
Training and test complexities of the proposed methods with single- and multi-core implementations
Method | Training complexity | Test complexity | |||
---|---|---|---|---|---|
Single-core | Multi-core | Single-core | Multi-core | ||
SST | true | \(O(m {\cdot } g_{tr}(n,d))\) | \(O(g_{tr}(n,d))\) | \(O(m {\cdot } g_{te}(n,d))\) | \(O(g_{te}(n,d))\) |
train | \(O(m {\cdot } g_{tr}(n,d))\) | \(O(g_{tr}(n,d))\) | \(O(m {\cdot } g_{te}(n,d))\) | \(O(g_{te}(n,d))\) | |
cv | \(O(f {\cdot } m {\cdot } g_{tr}(n,d))\) | \(O(g_{tr}(n,d))\) | \(O(m {\cdot } g_{te}(n,d))\) | \(O(g_{te}(n,d))\) | |
ERC | true | \(O(k {\cdot } m {\cdot } g_{tr}(n,d))\) | \(O(g_{tr}(n,d))\) | \(O(k {\cdot } m {\cdot } g_{te}(n,d))\) | \(O(m {\cdot } g_{te}(n,d))\) |
train | \(O(k {\cdot } m {\cdot } g_{tr}(n,d))\) | \(O( m {\cdot } g_{tr}(n,d))\) | \(O(k {\cdot } m {\cdot } g_{te}(n,d))\) | \(O(m {\cdot } g_{te}(n,d))\) | |
cv | \(O(k {\cdot } f {\cdot } m {\cdot } g_{tr}(n,d))\) | \(O( m {\cdot } g_{tr}(n,d))\) | \(O(k {\cdot } m {\cdot } g_{te}(n,d))\) | \(O(m {\cdot } g_{te}(n,d))\) |
Table 1 summarizes the training and test complexities of each method assuming a single-core implementation as well as the minimum possible complexity when a multi-core implementation is used. Note that, as shown in the table, SST\(_{cv}\) and ERC\(_{cv}\) have the same multi-core complexity with SST\(_{train}\) and ERC\(_{train}\) respectively because their internal cross-validation procedure can also be parallelized.
3 Related work
3.1 Multi-target regression
MTR was first studied in Statistics under the term multivariate regression with Reduced Rank Regression (RRR) (Izenman 1975), FICYREG (Merwe and Zidek 1980) and two-block PLS (Wold 1985) (the multiple response version of PLS) being three of the earliest methods. Among these methods, two-block PLS has been used more widely, especially in Chemometrics. More recently, the Curds and Whey (C&W) method was proposed (Breiman and Friedman 1997) and was found to outperform RRR, FICYREG and two-block PLS. As noted by Breiman and Friedman (1997), C&W, RRR and FICYREG can all be expressed using the same generic form \({\tilde{\mathbf{y}}= \mathbf{B} \hat{\mathbf{y}}}\), where \( {\hat{\mathbf{y}}}\) are estimates obtained by applying ordinary least squares regression on the target variables and \(\mathbf {B}\) is a matrix that modifies these estimates in order to obtain a more accurate prediction \({\tilde{\mathbf{y}}}\), under the assumption that the targets are correlated.
In all methods, \(\mathbf {B}\) can be expressed as \(\mathbf {B} = {\hat{\mathbf{T}}}^{-\mathbf{1}} \mathbf {D} {\hat{\mathbf{T}}}\), where \({\hat{\mathbf{T}}}\) is the matrix of sample canonical co-ordinates and \(\mathbf {D}\) is a diagonal “shrinking” matrix that is obtained differently in each method. SST is highly similar to these methods but allows a more general formulation of the MTR problem. Firstly, SST does not impose any restriction to the family of models that generate the uncorrected (first stage) estimates in contrast to these approaches that use estimates obtained from least squares regression. Secondly, the correction of the estimates applied by SST comes from a learning procedure that jointly considers target and input variables rather than target variables alone.
As shown by Breiman and Friedman (1997), the above methods can be described by an alternative but equivalent scheme. According to this, \(\mathbf {y}\) is first transformed to the canonical co-ordinate system \(\mathbf {y}' = {\hat{\mathbf{T}}} \mathbf {y}\), then separate least squares regression is performed on each \(\mathbf {y}'\) to obtain \({\hat{\mathbf{y}}'}\), these estimates are scaled by \(\mathbf {D}\) to obtain \({\tilde{\mathbf{y}}'} = \mathbf {D} {\hat{\mathbf{y}}'}\) and finally transformed back to the original output space \({\tilde{\mathbf{y}}} = {\hat{\mathbf{T}}^{-\mathbf{1}}} {\tilde{\mathbf{y}}'}\). As discussed by Dembczynski et al. (2012), from this perspective, these methods fall under a more general scheme where the output space is first transformed, single-target regressors are then trained on the transformed output space and an inverse transformation is performed (possibly along with shrinkage/regularization) to obtain predictions for the original targets. Due to its generality, this scheme has been adopted by a number of recent methods in both MLC (Hsu 2009; Zhang and Schneider 2011, 2012; Tai and Lin 2012) and MTR (Balasubramanian and Lebanon 2012; Tsoumakas et al. 2014).
A large number of MTR methods are derived from the predictive clustering tree (PCT) framework (Blockeel et al. 1998). The main difference between the PCT algorithm and a standard decision tree is that the variance and the prototype functions are treated as parameters that can be instantiated to fit the given learning task. Such an instantiation for MTR tasks are the multi-objective decision trees (MODTs) where the variance function is computed as the sum of the variances of the targets, and the prototype function is the vector mean of the target vectors of the training examples falling in each leaf (Blockeel et al. 1998, 1999). Bagging and random forest ensembles of MODTs were developed by Kocev et al. (2007) and were found significantly more accurate than MODTs and equally good or better than ensembles of single-objective decision trees for both regression and classification tasks. In particular, multi-objective random forests yielded better performance than multi-objective bagging.
Methods that deal with the prediction of multiple target variables can be found in the literature of the related learning task of multi-task learning. According to Caruana (1997), multi-task learning is a form of inductive transfer (Pratt 1992) where the aim is to improve generalization accuracy on a set of related tasks by using a shared representation that exploits commonalities between them. This definition implies that a multi-task method should be able to deal with problems where different prediction tasks do not necessarily share the same set of training examples or descriptive features and, moreover, each task can have a different data type. Thus, multi-task learning is actually a generalization of MTR.
Artificial neural networks (ANNs) are very well suited for multi-task problems because they can be naturally extended to support multiple outputs and offer flexibility in defining how inputs are shared between tasks. Thus, it is not surprising that most of the earliest multi-task methods were based on ANNs. Caruana (1994), for example, proposed a method where backpropagation is used to train single ANN with multiple outputs (connected to the same hidden layers), and showed that it has better generalization performance compared to multiple single-task ANNs. A different architecture was used by Baxter (1995) where only the first hidden layers are shared and subsequent layers are specific to each task. The question of how much sharing is better when multi-task ANNs are applied for stock return prediction was explored by Ghosn and Bengio (1996) who concluded that a partial sharing of network parameters is preferable compared to full or no sharing. More recently, Collobert and Weston (2008) applied a deep multi-task neural network architecture for natural language processing.
A large number of multi-task learning methods stem from a regularization perspective.^{6} Regularization-based multi-task methods minimize a penalized empirical loss of the form \(\displaystyle \min _W \mathscr {L}(W)+\varOmega (W)\), where W is a parameter matrix that has to be estimated, \(\mathscr {L}(W)\) is an empirical loss calculated on the training data and \(\varOmega (W)\) is a regularization term that takes a different form in each method depending on the underlying task relatedness assumption. Most methods assume that all tasks are related to each other (Evgeniou and Pontil 2004; Ando and Zhang 2005; Argyriou et al. 2006, 2008; Chen et al. 2009, 2010a; Obozinski et al. 2010), while there are methods assuming that tasks are organized in structures such as clusters (Jacob et al. 2008; Zhou et al. 2011a), trees (Kim and Xing 2010) and graphs (Chen et al. 2010b). A well-studied category of methods, which are particularly useful when dealing with high-dimensional input spaces, assume that models for different tasks share a common low-rank subspace and impose a trace-norm constraint on the parameter matrix (Argyriou et al. 2006, 2008; Ji and Ye 2009). A similar category of methods constraint all models to share a common set of features (thus performing a joint feature selection), typically by applying \(L_1 / L_q\)-norm (\(q>1\)) regularization (Obozinski et al. 2010). An approach that relaxes the above restrictive constraint allowing models to leverage different extents of feature sharing is proposed in Jalali et al. (2010, 2013).
Finally, we would like to mention that a number of MTR methods are based on the Gaussian Processes framework (e.g., Bonilla et al. 2007; Álvarez and Lawrence 2011). These methods capture correlations between tasks by appropriate choices of covariance functions. A nice review of such methods as well as their relations to regularization-based multi-task approaches can be found in Alvarez et al. (2011).
3.2 Discrepancy in meta-inputs
In the MLC domain, Senge et al. (2013a) studied how the discrepancy issue affects the performance of Classifier Chains and showed that longer chains (i.e. multi-label problems with more labels to be predicted) lead to a higher performance deterioration. In an extension of that work Senge et al. (2013b), a “rectified” version of Classifier Chains (called Nested Stacking) was presented that uses in-sample estimates of the label variables for training as in Stacked Binary Relevance. It was shown that this method performs better than the original Classifier Chains, especially when the label dependencies are strong. Following the opposite direction, Montañés et al. (2011) proposed AID, a method similar to Stacked Binary Relevance, and found that using the actual label values instead of (in-sample) estimates, leads to better results for most multi-label evaluation measures in both AID and Stacked Binary Relevance.
Our work is the first to study this issue in the MTR domain.^{7} The issue is studied jointly for SST and ERC, thus allowing general conclusions to be drawn for this type of methods. Furthermore, Montañés et al. (2011), Senge et al. (2013b) compared only the use of actual target values with the use of in-sample estimates while our comparison includes the use of out-of-sample estimates obtained by a cross-validation procedure. Finally, Senge et al. (2013b) evaluate the use of estimates in Classifier Chains whereas we focus on the ensemble version of the corresponding MTR method (ERC) that is expected to offer more resilience to error propagation, as discussed in Sect. 2.5.
4 Experimental setup
This section describes our experimental setup. We first present the participating methods and their parameters and provide details about their implementation in order to facilitate reproducibility of the experiments. Next, we describe the evaluation measure and explain the process that was followed for the statistical comparison of the methods. Finally, we present the datasets that we used and their main statistics.
4.1 Methods, parameters and implementation
Methods used in experiments with abbreviations and citations
Abbr. | Method | Citation |
---|---|---|
ST | Single target | |
SST\(_{true}\) | Stacked ST, true values | This paper |
SST\(_{train}\) | Stacked ST, in-sample estimates | This paper |
SST\(_{cv}\) | Stacked ST, cv estimates | This paper |
ERC\(_{true}\) | Ensemble of Regressor Chains, true values | This paper |
ERC\(_{train}\) | Ensemble of Regressor Chains, in-sample estimates | This paper |
ERC\(_{cv}\) | Ensemble of Regressor Chains, cv estimates | This paper |
MORF | Multi-Objective Random Forest | Kocev et al. (2007) |
TNR | Trace Norm Regularization multi-task learning | Argyriou et al. (2008) |
Dirty | A Dirty model for multi-task learning | |
RLC | Random Linear target Combinations | Tsoumakas et al. (2014) |
The proposed methods as well as ST and RLC transform the mutli-target regression task into a series of single-target regression tasks which can be dealt with using any standard regression algorithm. For most of the experiments, we use bagged regression trees as the base regressor. This choice was motivated in Sect. 2.3 and is further discussed in Sect. 5.1 where we present results using a variety of well-known linear and non-linear regression algorithms. The ensemble size of all ERC variants is set to \(k=10\) RC models, each one trained using a different random chain. In datasets with less than 10 distinct chains, we create exactly as many RC models as the number of distinct chains. Furthermore, since the base regressor involves bootstrap sampling, we do not perform sampling in ERC, i.e. each RC model is trained using all training examples. In SST, we exclude the target being predicted by each second stage model from the input space of that model as we found that this choice improves slightly the performance of all variants of this method. \(f=10\) internal cross-validation folds are used in both SST\(_{cv}\) and ERC\(_{cv}\).
Concerning the parameter settings of the competitive methods, in MORF we use an ensemble size of 100 trees and the values suggested by Kocev et al. (2007) for the rest of its parameters. In RLC, we generate \(r=100\) new target variables by combining \(k=2\) of the original target variables (after bringing them to the [0, 1] interval). As shown in Tsoumakas et al. (2014), these values lead to near optimal results. In TNR, we minimize the squared loss function using the accelerated gradient method for trace norm minimization (Ji and Ye 2009). The regularization parameter is tuned by selecting among the values \(\{10^r : r \in \{-3,\ldots ,3\}\}\) with internal 5-fold cross-validation. Before applying TNR, we apply z-score normalization and add a bias column as suggested in Zhou et al. (2011b). Finally, Dirty is setup as suggested in Jalali et al. (2013): Input variables are scaled to the \([-1,1]\) range by dividing them with their maximum values. The regularization parameters \(\lambda _b\) and \(\lambda _s\) are tuned via internal 5-fold cross-validation (as in TNR). As suggested in Jalali et al. (2013), we set \(\lambda _b = c \sqrt{\frac{m \log d}{n}}\), where \(c \in \{10^r : r \in \{-2,\ldots ,2\}\} \) is a constant. Each distinct value of \(\lambda _b\) is paired with five values of \(\lambda _s = \frac{\lambda _b}{1+ \frac{m-1}{4} i}, i \in \{0,1,2,3,4\}\), thus respecting the \(\frac{\lambda _s}{\lambda _b} \in [\frac{1}{m},1]\) relationship dictated by the optimality conditions. In total, 25 different combinations of \(\lambda _b\) and \(\lambda _s\) are evaluated.
All the proposed methods and the evaluation framework were implemented in Java and integrated in Mulan^{8} (Tsoumakas et al. 2011) by expanding its functionality to multi-target regression. The implementation of all single-target regression algorithms that were used to instantiate problem transformation methods are taken from Weka.^{9} With respect to the competing methods, RLC was already integrated in Mulan while for the purposes of this study we also integrated MORF (via a wrapper of the implementation offered in CLUS^{10}) as well as TNR and Dirty (via wrappers of the implementations offered in MALSAR (Zhou et al. 2011b)). Thus, all methods were evaluated under a common framework. In support of open science, we created a github project^{11} that contains all our implementations, including code that facilitates easy replication of our experimental results.
4.2 Evaluation
To test the statistical significance of the observed differences between the methods, we follow the methodology suggested by Demsar (2006). To compare multiple methods on multiple datasets we use the Friedman test, the non-parametric alternative of the repeated-measures ANOVA. The Friedman test operates on the average ranks of the methods and checks the validity of the hypothesis (null-hypothesis) that all methods are equivalent. Here, we use an improved (less conservative) version of the test that uses the \(F_f\) instead of the \(\chi ^2_F\) statistic (Iman and Davenport 1980). When the null-hypothesis of the Friedman test is rejected (\(p < 0.01\)), we proceed with the Nemenyi post-hoc test that compares all methods to each other in order to find which methods in particular differ from each other. Instead of reporting the outcomes of all pairwise comparisons, we employ the simple graphical presentation of the test’s results introduced by Demsar (2006), i.e. all methods being compared are placed in a horizontal axis according to their average ranks and groups of methods that are not significantly different (at a certain significance level) are connected (see Fig. 4 for an example). To generate such a diagram, a critical difference (CD) should be calculated that corresponds to the minimum difference in average ranks required for two methods to be considered significantly different. CD for a given number of methods and datasets, depends on the desired significance level. Due to the known conservancy of the Nemenyi test (Demsar 2006), we use a 0.05 significance level for computing the CD throughout the paper.
As the above methodology requires a single performance measurement for each method on each dataset, it is not directly applicable to multi-target evaluation where we have multiple performance measurements (one for each target) for each method on each dataset. One option is to take the average RRMSE (aRRMSE) across all target variables within a dataset as a single performance measurement. This choice, however, has the disadvantage that a very small or large error on a single target might dominate the average, thus obscuring performance differences on the target level. Another option is to treat the RRMSE of each method on each target as a different performance measurement. In this case, Friedman test’s assumption of independence between performance measurements might be violated. In the absence of a better solution, we perform a two-dimensional analysis (as done e.g. by Aho et al. 2012) where statistical tests are conducted using both aRRMSE (per dataset analysis) but also considering RRMSE per target as an independent performance measurement (per target analysis).
4.3 Datasets
Name, source, number of examples, number of input variables (d) and number of target variables (m) of the datasets used in the evaluation
Dataset | Source | Examples | d | m |
---|---|---|---|---|
edm | Karalic and Bratko (1997) | 154 | 16 | 2 |
sf1 | Lichman (2013) | 323 | 10 | 3 |
sf2 | Lichman (2013) | 1066 | 10 | 3 |
jura | Goovaerts (1997) | 359 | 15 | 3 |
wq | Dzeroski et al. (2000) | 1060 | 16 | 14 |
enb | Tsanas and Xifara (2012) | 768 | 8 | 2 |
slump | Yeh (2007) | 103 | 7 | 3 |
andro | Hatzikos et al. (2008) | 49 | 30 | 6 |
osales | Kaggle (2012) | 639 | 413 | 12 |
scpf | Kaggle (2013) | 1137 | 23 | 3 |
atp1d | This paper | 337 | 411 | 6 |
atp7d | This paper | 296 | 411 | 6 |
oes97 | This paper | 334 | 263 | 16 |
oes10 | This paper | 403 | 298 | 16 |
rf1 | This paper | 9125 | 64 | 8 |
rf2 | This paper | 9125 | 576 | 8 |
scm1d | This paper | 9803 | 280 | 16 |
scm20d | This paper | 8966 | 61 | 16 |
Table 3 reports the name (1st column), source (2nd column), number of examples (3rd column), number of input variables (4th column) and number of target variables (5th column) of each dataset. Detailed descriptions of all datasets are provided in “Appendix”.
5 Experimental analysis
In this section we present an extensive experimental analysis of the performance of the proposed methods. Sect. 5.1 is devoted to an exploration of the performance of ST using various well-known regression algorithms. The purpose of this investigation is to help us select an algorithm that works well on the studied datasets and use it as base regressor in all problem transformation methods (ST, SST, ERC and RLC) in subsequent experiments. At the same time, a challenging baseline performance level will be set for all multi-target methods. In Sect. 5.2 we evaluate SST\(_{train}\) and ERC\(_{true}\), the direct adaptations of the corresponding MLC methods, in order to see whether these variants obtain a competitive performance compared to ST and state-of-the-art multi-target methods. Next, in Sect. 5.3 all three meta-input generation variants (true, train, cv) of SST and ERC are evaluated and compared to ST, shedding light into the impact of the discrepancy problem on each method. After the best performing variants of each method have been identified, Sect. 5.4 compares them with the state-of-the-art. The running times of all methods are reported and compared in Sect. 5.5, and finally, this section ends with a discussion of the main outcomes of the experimental results (Sect. 5.6).
5.1 Base regressor exploration
In this subsection we explore the performance of ST on the studied domains using a variety of regression algorithms. The goal of this exploration is to help us identify a regression algorithm that performs well across many domains, thus setting a challenging baseline performance level for the multi-target methods that we study next. The algorithm that will emerge as the best performer will be used to instantiate all problem transformation methods (ST, SST, ERC and RLC) in the rest of the experiments, facilitating a fair comparison between these methods.
We selected five well-known linear and non-linear regression algorithms to couple ST with, in particular we use: ridge regression (Hoerl and Kennard 1970) (ridge), regression tree Breiman et al. (1984) (tree), L2-regularized support vector regression regression (Drucker et al. 1996) (svr), bagged (Breiman 1996) regression trees (bag) and stochastic gradient boosting (Friedman 2002) (sgb). In ridge and svr, the regularization parameter was tuned (separately for each target) by applying internal 5-fold cross-validation and choosing the value that leads to the lowest root mean squared error among \(\{10^r : r \in \{-4,\ldots ,2\}\}\). In bag we combine the predictions of 100 trees while in sgb we boost trees with four terminal nodes using a small shrinkage rate (0.1) and a large number of iterations (100), as suggested by Friedman et al. (2001).
5.2 Evaluation of direct adaptations
In this subsection we focus on SST\(_{train}\) and ERC\(_{true}\), the versions of SST and ERC that use the same type of values for the meta-inputs as their MLC counterparts, and compare their performance to that of ST, MORF, RLC, TNR and Dirty to see where these methods stand with respect to the state-of-the-art.
Interestingly, however, we see that according to both the per dataset and the per target analysis, SST\(_{train}\) and ERC\(_{true}\) are not significantly better than ST. This is an indication that the use of targets as meta-input as implemented by these variants of SST and ERC does not bring significant improvements. Actually, as can be seen from the detailed results, both SST\(_{train}\) and ERC\(_{true}\) perform worse than ST in several cases. This issue is studied in more detail in the following subsection.
Perhaps even more interestingly, none of the state-of-the-art multi-target methods participating in this comparison manages to significantly improve the performance of ST. In fact, ST is ranked second after SST\(_{train}\) in the per dataset analysis and third after SST\(_{train}\) and RLC in the per target analysis, and is found significantly better than TNR and Dirty in both types of analyses. This exceptionally good performance of ST might seem a bit surprising given the results of previous studies (e.g. Kocev et al. 2007; Tsoumakas et al. 2014) but is in accordance with empirical and theoretical results for Binary Relevance (as discussed in Sect. 2) and is attributed to the use of a very strong base regressor.
5.3 Evaluation of meta-input generation variants
Figures 6 and 7 show the average ranks and the results of the Friedman and Nemenyi tests for the three variants of SST and ERC, respectively, according to the per dataset (left) and the per target (right) analysis. First, we see that in both SST and ERC and in both types of analyses, the variants that use the actual values of the targets (true) obtain the worst average ranks and are found significantly worse than both variants that use estimates (train and cv). Since the variants of each method differ only with respect to the type of values that they use for the meta-inputs, it is clear that the discrepancy problem has a significant impact on the performance of both SST and ERC and that the use of estimates can ameliorate this problem.
5.3.1 Cautiousness analysis
So far, our analysis has focused on the average performance of the proposed methods (as quantified by their average ranks over datasets and targets) and we found that ERC\(_{train}\) and ERC\(_{cv}\) outperform the independent regressions baseline significantly. However, it is also important to see the consistency of these improvements across different datasets and targets. In particular, we would like to study the degree of cautiousness that each method exhibits, i.e. how frequently and to what extent are the predictions produced by each method less accurate than the predictions of ST.
We see that that in both the per dataset and the per target analysis, the true variants are the ones exhibiting the more dispersed distributions with several cases of significant degradation of ST’s performance. The train and cv variants are clearly more cautious with much fewer cases of degradation and even fewer cases of significant degradation. Looking at the distributions of \(R_t\), we could say that thecvvariants appear a bit more cautious than thetrainvariants especially in the case of SST. We also see that the ERC variants are always more cautious than the corresponding SST variants. Clearly, ERC\(_{train}\) and ERC\(_{cv}\) are the two most cautious methods since they obtain very similar or better performance than ST on all datasets and on about 75 % of the targets. Even on targets where the two methods obtain a lower performance than ST, the reduction is less than about 5 %. This characteristic along with the fact that they obtain the largest average improvements over ST, make ERC\(_{train}\) and ERC\(_{cv}\) highly appealing.
5.4 Comparison with the state-of-the-art
5.5 Running times
In this subsection we compare the running times of the studied methods. Experiments were run on a 64-bit CentOS Linux machine with 80 Intel Xeon E7-4860 processors running at 2.27 GHz and 1 TB of main memory. The detailed results per method and dataset are shown in Table 4. For ST, RLC, SST and ERC we report times with bag as base regressor. The number shown in parenthesis next to the name of each dataset corresponds to the maximum number of processor threads that were available during the experiment. ST, SST, ERC and RLC made use of multiple threads through Weka’s multi-threaded implementation of Bagging. Thus, running times are directly comparable for these methods. Multi-threading was also partly used in TNR for the computation of the gradients. dirty and MORF, on the other hand, always used a single processor thread.
Looking at the aggregated running times, we see that MORF is by far the most efficient method, followed by ST, SST\(_{true}\) and SST\(_{train}\) which have similar running times. On the other hand, dirty is the least efficient method, followed by ERC\(_{cv}\). The running times of the rest of the methods lie in between. With respect to the SST and ERC variants, we see that their running times agree with the complexity analysis of Sect. 2.6. The total running time of SST\(_{true}\) is roughly twice the total running time of ST and similar to the total running time of SST\(_{train}\). SST\(_{cv}\) is the least efficient among SST variants with a total running time that is about 5 times larger than that of SST\(_{true}\) and SST\(_{train}\). With respect to the ERC variants, we see that ERC\(_{true}\) and ERC\(_{train}\) have similar total running times (which are also roughly similar to the total running time of SST\(_{cv}\)) while ERC\(_{cv}\) is about 7.5 times slower.
Running times (in s) using bag as base regressor
Dataset | ST | SST\(_{true}\) | SST\(_{train}\) | SST\(_{cv}\) | ERC\(_{true}\) | ERC\(_{train}\) | ERC\(_{cv}\) | MORF | RLC | TNR | dirty |
---|---|---|---|---|---|---|---|---|---|---|---|
edm (2) | 4.3 | 3.3 | 3.2 | 14.5 | 2.8 | 2.8 | 13.7 | 6.3 | 78.3 | 16.6 | 281.4 |
sf1 (3) | 4.1 | 4.1 | 4.3 | 16.7 | 11.2 | 11.8 | 60.8 | 8.1 | 79.2 | 11.8 | 60.5 |
sf2 (3) | 7.5 | 11.5 | 11.4 | 47.6 | 33.2 | 36.3 | 262.0 | 15.4 | 238.7 | 17.3 | 56.9 |
jura (3) | 10.9 | 15.3 | 14.9 | 72.6 | 46.4 | 46.9 | 264.2 | 14.0 | 199.1 | 17.0 | 49.2 |
wq (6) | 43.3 | 104.0 | 141.7 | 540.0 | 561.3 | 607.0 | 4648.4 | 106.3 | 295.0 | 143.7 | 531.2 |
enb (2) | 10.4 | 15.4 | 15.9 | 76.0 | 16.7 | 16.4 | 74.9 | 15.7 | 313.9 | 35.8 | 223.0 |
slump (3) | 4.7 | 4.2 | 4.0 | 16.5 | 11.4 | 11.1 | 67.8 | 4.2 | 48.5 | 10.7 | 55.6 |
andro (2) | 8.2 | 10.6 | 9.0 | 47.9 | 49.5 | 47.2 | 360.3 | 3.1 | 72.5 | 445.3 | 1670.8 |
onsales (8) | 317.6 | 612.2 | 560.8 | 2803.0 | 2835.0 | 2801.6 | 21568.6 | 43.2 | 1628.0 | 2769.1 | 80616.2 |
scpf (3) | 22.9 | 36.2 | 36.6 | 196.5 | 106.2 | 106.4 | 651.2 | 13.4 | 491.3 | 84.5 | 215.4 |
atp1d (4) | 155.6 | 356.1 | 345.7 | 1614.7 | 1496.0 | 1555.6 | 12169.2 | 23.3 | 2408.0 | 2212.8 | 57276.5 |
atp7d (4) | 145.8 | 318.6 | 313.9 | 1517.2 | 1336.9 | 1385.0 | 11158.2 | 18.4 | 2179.2 | 1536.0 | 53974.0 |
oes97 (6) | 199.5 | 454.8 | 422.2 | 1998.4 | 1977.3 | 1965.5 | 16609.1 | 32.7 | 1057.1 | 4581.3 | 124231.5 |
oes10 (6) | 286.4 | 613.1 | 535.7 | 2725.9 | 2691.0 | 2575.1 | 21446.6 | 39.4 | 1182.6 | 6093.8 | 157399.6 |
rf1 (8) | 379.6 | 868.2 | 890.8 | 4180.7 | 3885.7 | 4018.8 | 30050.1 | 351.1 | 4184.7 | 2952.1 | 34927.2 |
rf2 (10) | 3505.5 | 6539.0 | 6065.6 | 31562.8 | 28347.1 | 28104.0 | 234088.5 | 325.2 | 35429.1 | 18130.9 | 197482.1 |
scm1d (10) | 1398.5 | 2499.5 | 2398.3 | 11111.9 | 10465.6 | 11253.2 | 113500.0 | 199.7 | 6018.6 | 5647.1 | 105215.8 |
scm20d (8) | 302.6 | 613.9 | 621.8 | 2615.8 | 2733.1 | 2646.9 | 18549.2 | 140.8 | 1627.4 | 1120.2 | 4779.6 |
Total | 6807.6 | 13079.8 | 12395.8 | 61158.7 | 56606.3 | 57191.6 | 485542.9 | 1360.2 | 57531.3 | 45826.0 | 819046.5 |
5.6 Discussion
Several interesting conclusions can be drawn from our experimental results. The experiments of Sects. 5.2 and 5.3 showed that while the directly adapted versions of SST and ERC have comparable or better performance than state-of-the-art methods, a careful handling of the discrepancy problem is crucial for obtaining consistent improvements over the independent regressions baseline and the state-of-the-art. In particular, as the experiments of Sect. 5.3 revealed, the use of estimates for the meta-inputs during training should clearly be preferred over using the actual target values. With regard to using in-sample versus out-of-sample estimates, the results indicate that while out-of-sample estimates are preferable in SST, ERC performs almost equally well using either type of estimates for the meta-inputs. As discussed in Sect. 2.5, ERC’s models are built on input spaces which are expanded with fewer meta-inputs compared to SST’s models and, as a result, a smaller amount of error accumulation is risked at prediction time.
Another interesting conclusion is that when a strong base regressor is employed, the task of improving the performance of ST becomes very difficult. As a result, multi-target methods which are considered state-of-the-art fail to improve ST’s performance and are even performing significantly worse. This was particularly the case for the two multi-task methods, TNR and Dirty, which were consistently found to be the worst performers. One explanation for their bad performance is the fact that both methods are based on a linear formulation of the problem that, as revealed by the base regressor exploration experiments, is not the most suitable hypothesis representation for the studied datasets (ridge and svr performed worse than sgb and bag that are based on a non-linear hypothesis representation). Moreover, multi-task methods are expected to work better than single-task methods in cases where there is a lack of training data for some of the tasks (Alvarez et al. 2011). This is not the case for most of the datasets that we used in this study as well as many recent multi-target prediction problems. In fact, the two datasets where TNR and Dirty perform better than ST (sf1 and slump) are among those with the fewest training examples.
With respect to MORF, although it was found significantly more competitive than TNR and Dirty, it also performed worse than ST on average. Nevertheless, we should point out that MORF achieved the best accuracy on three datasets (edm, wq, andro) and is the most computationally efficient of the compared methods. Similarly to TNR and Dirty, MORF has the disadvantage of having a fixed hypothesis representation (trees), as opposed to the proposed methods that have the ability of adapting better to a specific domain by being instantiated with a more suitable base regressor. This advantage of the proposed methods is shared with RLC which, however, was not found as accurate.
Overall, our experimental results demonstrate that of the methods proposed in this paper, ERC\(_{train}\) and ERC\(_{cv}\) and, to a lesser extent, SST\(_{train}\) and SST\(_{cv}\) provide increased accuracy over doing a separate regression per target. In addition, ERC\(_{train}\) and ERC\(_{cv}\) are significantly more accurate than TNR, Dirty, MORF and RLC (in the per target analysis). If caution is a further concern, then again ERC\(_{train}\) and ERC\(_{cv}\) compare favorably to the rest of the methods. With respect to the true variants of SST and ERC, we should stress out that despite having a worse average performance, they are worthy of being considered by a practitioner as they obtain the highest performance in datasets (e.g., sf1 and scfp) where the discrepancy problem is not predominant.
6 Conclusion
Motivated by the similarity between the tasks of multi-label classification and multi-target regression, this paper introduced SST and ERC, two new multi-target regression techniques derived through a simple adaptation of two well-known multi-label classification methods. Both methods are based on the idea of treating other prediction targets as additional input variables, and represent a conceptually simple way of exploiting target dependencies in order to improve prediction accuracy.
A comprehensive experimental analysis that includes a multitude of real-world datasets and four existing state-of-the-art methods, reveals that, despite being competitive with the state-of-the-art, the directly adapted versions of SST and ERC do not manage to obtain significant improvements or even degrade the performance of the independent regressions baseline. This degradation is attributed to an underestimation (in the original formulations of the methods) of the impact of the discrepancy of the values used for the additional input variables between training and prediction. Confirming our hypothesis, extended versions of the methods that attempt to mitigate the discrepancy using out-of-sample estimates of the targets during training, manage to obtain consistent and significant improvements over the baseline approach and are found significantly better than four state-of-the-art methods. The fact that these impressive results were obtained by applying relatively simple adaptations of existing multi-label classification methods, highlights the importance of exploiting relationships between similar machine learning tasks.
Concluding, let us point to some directions for future work. Although a mitigation of the discrepancy problem leads to significant performance improvements, a different amount of mitigation is ideal for each target. As a result, the use of in-sample estimates (or even the actual target values) gives better results for some targets. Thus, a promising direction for future work would be a deeper theoretical analysis of the different variants and the identification of problem characteristics that favor the use of one variant over the other. Finally, we should point out that SST and ERC can be viewed as strategies for leveraging variables that are available in the training phase but not in the prediction phase. This type of scenario is very common, for instance, in time series prediction. We believe that adapting SST and ERC for this type of problems is another valuable opportunity for future work.
Footnotes
- 1.
See NIPS’11 workshop on relations among machine learning problems at http://rml.anu.edu.au/.
- 2.
\(\mathscr {X}=R^d\) is used only for the sake of brevity. The domain of the input variables can also be discrete.
- 3.
Note, however, that this analysis concerns a version of stacking that does not include the original input variables in the input space of the second stage models.
- 4.
This is in contrast with traditional deep learning where high-level feature representations are typically learned from the data in an unsupervised way.
- 5.
An explicit feature selection could alternatively be applied as a means of variance reduction.
- 6.
A nice categorization of regularization-based multi-task methods can be found in Zhou et al. (2012).
- 7.
Actually, an early version of this work Spyromitros-Xioufis et al. (2012) is the first to consider the discrepancy problem in the context of input space expansion methods.
- 8.
- 9.
- 10.
- 11.
- 12.
The reliability of the estimates obtained using \(k=2\) and \(k=5\) has been validated by checking the stability of the rankings of the methods when repeating the cross-validation experiment with different random seeds.
- 13.
- 14.
The detailed results per dataset and target can be found in Appendix “Multi-target regression results”.
References
- Aho, T., Zenko, B., Dzeroski, S., & Elomaa, T. (2012). Multi-target regression with rule ensembles. Journal of Machine Learning Research, 13, 2367–2407.MathSciNetMATHGoogle Scholar
- Álvarez, M. A., & Lawrence, N. D. (2011). Computationally efficient convolved multiple output gaussian processes. Journal of Machine Learning Research, 12, 1459–1500.MathSciNetMATHGoogle Scholar
- Alvarez, M. A., Rosasco, L., & Lawrence, N. D. (2011). Kernels for vector-valued functions: A review. arXiv preprint arXiv:1106.6251.
- Ando, R. K., & Zhang, T. (2005). A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6, 1817–1853.MathSciNetMATHGoogle Scholar
- Argyriou, A., Evgeniou, T., & Pontil, M. (2006). Multi-task feature learning. In Advances in neural information processing systems 19, Proceedings of the twentieth annual conference on neural information processing systems, Vancouver, British Columbia, Canada, December 4–7, 2006, pp. 41–48.Google Scholar
- Argyriou, A., Evgeniou, T., & Pontil, M. (2008). Convex multi-task feature learning. Machine Learning, 73(3), 243–272.CrossRefGoogle Scholar
- Balasubramanian, K., & Lebanon, G. (2012). The landmark selection method for multiple output prediction. In Proceedings of the 29th international conference on machine learning, ICML 2012, Edinburgh, Scotland, UK, June 26–July 1, 2012.Google Scholar
- Baxter, J. (1995). Learning internal representations. In Proceedings of the eigth annual conference on computational learning theory, COLT 1995, Santa Cruz, California, USA, July 5–8, 1995, pp. 311–320.Google Scholar
- Blockeel, H., Raedt, L. D., & Ramon, J. (1998). Top-down induction of clustering trees. In Proceedings of the fifteenth international conference on machine learning (ICML 1998), Madison, Wisconsin, USA, July 24–27, 1998, pp. 55–63.Google Scholar
- Blockeel, H., Dzeroski, S., & Grbovic, J. (1999). Simultaneous prediction of mulriple chemical parameters of river water quality with TILDE. In Proceedings of third European conference in principles of data mining and knowledge discovery PKDD’99, Prague, Czech Republic, September 15–18, 1999, pp. 32–40.Google Scholar
- Bonilla, E. V., Chai, K. M. A., & Williams, C. K. I. (2007). Multi-task gaussian process prediction. In Advances in neural information processing systems 20, proceedings of the twenty-first annual conference on neural information processing systems, Vancouver, British Columbia, Canada, December 3–6, 2007, pp. 153–160.Google Scholar
- Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140.MathSciNetMATHGoogle Scholar
- Breiman, L., & Friedman, J. H. (1997). Predicting multivariate responses in multiple linear regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 59(1), 3–54.MathSciNetCrossRefMATHGoogle Scholar
- Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and regression trees. Belmont, CA: Wadsworth.MATHGoogle Scholar
- Caruana, R. (1994). Learning many related tasks at the same time with backpropagation. In Advances in neural information processing systems 7, [NIPS Conference, Denver, Colorado, USA, 1994], pp. 657–664.Google Scholar
- Caruana, R. (1997). Multitask learning. Machine Learning, 28(1), 41–75.MathSciNetCrossRefGoogle Scholar
- Chen, J., Tang, L., Liu, J., & Ye, J. (2009). A convex formulation for learning shared structures from multiple tasks. In Proceedings of the 26th annual international conference on machine learning, ICML 2009, Montreal, Quebec, Canada, June 14–18, 2009, pp. 137–144.Google Scholar
- Chen, J., Liu, J., & Ye, J. (2010a). Learning incoherent sparse and low-rank patterns from multiple tasks. In Proceedings of the 16th ACM SIGKDD international conference on knowledge discovery and data mining, Washington, DC, USA, July 25–28, 2010, pp. 1179–1188.Google Scholar
- Chen, X., Kim, S., Lin, Q., Carbonell, J. G., & Xing, E. P. (2010b). Graph-structured multi-task regression and an efficient optimization method for general fused lasso. arXiv preprint arXiv:1005.3579.
- Cheng, W., & Hüllermeier, E. (2009). Combining instance-based learning and logistic regression for multilabel classification. Machine Learning, 76(2–3), 211–225.CrossRefGoogle Scholar
- Collobert, R., & Weston, J. (2008). A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of the twenty-fifth international conference on machine learning, (ICML 2008), Helsinki, Finland, June 5–9, 2008, pp. 160–167.Google Scholar
- Dembczynski, K., Cheng, W., & Hüllermeier, E. (2010). Bayes optimal multilabel classification via probabilistic classifier chains. In Proceedings of the 27th international conference on machine learning (ICML-10), June 21-24, 2010, Haifa, Israel, pp. 279–286.Google Scholar
- Dembczynski, K., Waegeman, W., Cheng, W., & Hüllermeier, E. (2012). On label dependence and loss minimization in multi-label classification. Machine Learning, 88(1–2), 5–45.MathSciNetCrossRefMATHGoogle Scholar
- Demsar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30.MathSciNetMATHGoogle Scholar
- Drucker, H., Burges, C. J. C., Kaufman, L., Smola, A. J., & Vapnik, V. (1996). Support vector regression machines. In Advances in neural information processing systems 9, NIPS, Denver, CO, USA, December 2–5, 1996, pp. 155–161.Google Scholar
- Dzeroski, S., Demsar, D., & Grbovic, J. (2000). Predicting chemical parameters of river water quality from bioindicator data. Applied Intelligence, 13(1), 7–17.CrossRefGoogle Scholar
- Evgeniou, T., & Pontil, M. (2004). Regularized multi-task learning. In Proceedings of the tenth ACM SIGKDD international conference on knowledge discovery and data mining, Seattle, Washington, USA, August 22–25, 2004, pp. 109–117.Google Scholar
- Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning. Berlin: Springer series in statistics Springer.MATHGoogle Scholar
- Friedman, J. H. (2002). Stochastic gradient boosting. Computational Statistics & Data Analysis, 38(4), 367–378.MathSciNetCrossRefMATHGoogle Scholar
- Ghosn, J., & Bengio, Y. (1996). Multi-task learning for stock selection. In Advances in neural information processing systems 9, NIPS, Denver, CO, USA, December 2–5, 1996, pp. 946–952.Google Scholar
- Godbole, S., & Sarawagi, S. (2004). Discriminative methods for multi-labeled classification. In Proceedings of 8th Pacific-Asia conference on advances in knowledge discovery and data mining, PAKDD 2004, Sydney, Australia, May 26–28, 2004, pp. 22–30.Google Scholar
- Goovaerts, P. (1997). Geostatistics for natural resources evaluation. Oxford: Oxford University Press.Google Scholar
- Groves, W., & Gini, M. L. (2011). Improving prediction in TAC SCM by integrating multivariate and temporal aspects via PLS regression. In Agent-mediated electronic commerce. Designing trading strategies and mechanisms for electronic markets - AMEC 2011, Taipei, Taiwan, May 2, 2011, and TADA 2011, Barcelona, Spain, July 17, 2011, Revised Selected Papers, pp. 28–43.Google Scholar
- Groves, W., & Gini, M. L. (2015). On optimizing airline ticket purchase timing. ACM Transactions on Intelligent Systems and Technology (TIST), 7(1), 3.Google Scholar
- Hatzikos, E. V., Tsoumakas, G., Tzanis, G., Bassiliades, N., & Vlahavas, I. P. (2008). An empirical study on sea water quality prediction. Knowledge-Based Systems, 21(6), 471–478.CrossRefGoogle Scholar
- Hoerl, A. E., & Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67.MathSciNetCrossRefMATHGoogle Scholar
- Hsu, D., Kakade, S., Langford, J., & Zhang, T. (2009). Multi-label prediction via compressed sensing. In Advances in neural information processing systems 22: 23rd annual conference on neural information processing systems 2009. Proceedings of a meeting held 7–10 December 2009, Vancouver, British Columbia, Canada, pp. 772–780.Google Scholar
- Iman, R. L., & Davenport, J. M. (1980). Approximations of the critical region of the friedman statistic. Communications in Statistics-Theory and Methods, 9(6), 571–595.CrossRefMATHGoogle Scholar
- Izenman, A. J. (1975). Reduced-rank regression for the multivariate linear model. Journal of Multivariate Analysis, 5(2), 248–264.MathSciNetCrossRefMATHGoogle Scholar
- Izenman, A. J. (2008). Modern multivariate statistical techniques: Regression, classification, and manifold learning. New York: Springer.CrossRefMATHGoogle Scholar
- Jacob, L., Bach, F. R., & Vert, J. (2008). Clustered multi-task learning: A convex formulation. In Advances in neural information processing systems 21. Proceedings of the twenty-second annual conference on neural information processing systems, Vancouver, British Columbia, Canada, December 8–11, 2008, pp 745–752.Google Scholar
- Jalali, A., Ravikumar, P. D., Sanghavi, S., & Ruan, C. (2010) A dirty model for multi-task learning. In Advances in neural information processing systems 23: 24th annual conference on neural information processing systems 2010. Proceedings of a meeting held 6–9 December 2010, Vancouver, British Columbia, Canada, pp. 964–972.Google Scholar
- Jalali, A., Ravikumar, P. D., & Sanghavi, S. (2013). A dirty model for multiple sparse regression. IEEE Transactions on Information Theory, 59(12), 7947–7968.MathSciNetCrossRefGoogle Scholar
- Ji, S., & Ye, J. (2009). An accelerated gradient method for trace norm minimization. In Proceedings of the 26th annual international conference on machine learning, ICML 2009, Montreal, Quebec, Canada, June 14–18, 2009, pp. 457–464.Google Scholar
- Kaggle. (2012). Kaggle competition: Online product sales. https://www.kaggle.com/c/online-sales
- Kaggle. (2013). Kaggle competition: See click predict fix. https://www.kaggle.com/c/see-click-predict-fix
- Karalic, A., & Bratko, I. (1997). First order regression. Machine Learning, 26(2–3), 147–176.CrossRefMATHGoogle Scholar
- Kim, S., & Xing, E.P. (2010). Tree-guided group lasso for multi-task regression with structured sparsity. In Proceedings of the 27th international conference on machine learning (ICML-10), June 21–24, 2010, Haifa, Israel, pp. 543–550.Google Scholar
- Kocev, D., Vens, C., Struyf, J., & Dzeroski, S. (2007). Ensembles of multi-objective decision trees. In Proceedings of 18th European conference on machine learning: ECML 2007, Warsaw, Poland, September 17–21, 2007, pp. 624–631.Google Scholar
- Kocev, D., Džeroski, S., White, M. D., Newell, G. R., & Griffioen, P. (2009). Using single-and multi-target regression trees and ensembles to model a compound index of vegetation condition. Ecological Modelling, 220(8), 1159–1168.CrossRefGoogle Scholar
- Kumar, A., Vembu, S., Menon, A. K., & Elkan, C. (2012). Learning and inference in probabilistic classifier chains with beam search. In Proceedings of European conference on machine learning and knowledge discovery in databases, Part I, ECML PKDD 2012, Bristol, UK, September 24–28, 2012. pp. 665–680.Google Scholar
- Lichman, M. (2013). UCI machine learning repository. http://archive.ics.uci.edu/ml
- Luaces, O., Díez, J., Barranquero, J., del Coz, J. J., & Bahamonde, A. (2012). Binary relevance efficacy for multilabel classification. Progress in Artificial Intelligence, 1(4), 303–313.CrossRefGoogle Scholar
- Montañés, E., Quevedo, J. R., & del Coz, J. J. (2011). Aggregating independent and dependent models to learn multi-label classifiers. In Proceedings of European conference on machine learning and knowledge discovery in databases, Part II, ECML PKDD 2011, Athens, Greece, September 5–9, 2011, pp. 484–500.Google Scholar
- Munson, M. A., & Caruana, R. (2009). On feature selection, bias-variance, and bagging. In Proceedings of European conference on machine learning and knowledge discovery in databases, Part II, ECML PKDD 2009, Bled, Slovenia, September 7–11, 2009, pp. 144–159.Google Scholar
- Obozinski, G., Taskar, B., & Jordan, M. I. (2010). Joint covariate selection and joint subspace selection for multiple classification problems. Statistics and Computing, 20(2), 231–252.MathSciNetCrossRefGoogle Scholar
- Pardoe, D., & Stone, P. (2008). The 2007 TAC SCM prediction challenge. In AAAI 2008 workshop on trading agent design and analysis.Google Scholar
- Pratt, L. Y. (1992). Discriminability-based transfer between neural networks. In Advances in neural information processing systems 5, [NIPS Conference, Denver, Colorado, USA, November 30–December 3, 1992], pp. 204–211.Google Scholar
- Read, J., & Hollmén, J. (2014). A deep interpretation of classifier chains. In Proceedings of 13th international symposium on advances in intelligent data analysis XIII, IDA 2014, Leuven, Belgium, October 30–November 1, 2014, pp. 251–262.Google Scholar
- Read, J., & Hollmén, J. (2015). Multi-label classification using labels as hidden nodes. arXiv preprint arXiv:1503.09022.
- Read, J., Pfahringer, B., Holmes, G., & Frank, E. (2011). Classifier chains for multi-label classification. Machine Learning, 85(3), 333–359.MathSciNetCrossRefGoogle Scholar
- Read, J., Martino, L., & Luengo, D. (2014). Efficient monte carlo methods for multi-dimensional learning with classifier chains. Pattern Recognition, 47(3), 1535–1546.CrossRefMATHGoogle Scholar
- Senge, R., del Coz, J. J., & Hüllermeier, E. (2013a) On the problem of error propagation in classifier chains for multi-label classification. In Proceedings of the 36th annual conference of the German classification society.Google Scholar
- Senge, R., del Coz, J. J., & Hüllermeier, E. (2013b) Rectifying classifier chains for multi-label classification. In LWA 2013. Lernen, Wissen & Adaptivität, workshop proceedings Bamberg, 7–9 October 2013, pp. 151–158.Google Scholar
- Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W., & Vlahavas, I. (2012). Multi-label classification methods for multi-target regression. ArXiv e-prints arXiv:1211.6581v1.
- Su, J., & Zhang, H. (2006) A fast decision tree learning algorithm. In Proceedings, the twenty-first national conference on artificial intelligence and the eighteenth innovative applications of artificial intelligence conference, July 16–20, 2006, Boston, Massachusetts, USA, pp 500–505Google Scholar
- Tai, F., & Lin, H. (2012). Multilabel classification with principal label space transformation. Neural Computation, 24(9), 2508–2542.MathSciNetCrossRefMATHGoogle Scholar
- Teh, Y. W., Seeger, M., & Jordan, M. I. (2005). Semiparametric latent factor models. In Proceedings of the tenth international workshop on artificial intelligence and statistics, AISTATS 2005, Bridgetown, Barbados, January 6–8, 2005.Google Scholar
- Tsanas, A., & Xifara, A. (2012). Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools. Energy and Buildings, 49, 560–567.CrossRefGoogle Scholar
- Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In O. Maimon & L. Rokach (Eds.), Data mining and knowledge discovery handbook (2nd ed., pp. 667–685). Boston, MA: Springer.Google Scholar
- Tsoumakas, G., Spyromitros-Xioufis, E., Vilcek, J., & Vlahavas, I. (2011). Mulan: A java library for multi-label learning. Journal of Machine Learning Research, 12, 2411–2414.MathSciNetMATHGoogle Scholar
- Tsoumakas, G., Xioufis, E.S., Vrekou, A., & Vlahavas, I. P. (2014). Multi-target regression via random linear target combinations. In Proceedings of European conference on machine learning and knowledge discovery in databases, Part III, ECML PKDD 2014, Nancy, France, September 15–19, 2014, pp. 225–240.Google Scholar
- Van Der Merwe, A., & Zidek, J. (1980). Multivariate regression analysis and canonical variates. Canadian Journal of Statistics, 8(1), 27–39.MathSciNetCrossRefMATHGoogle Scholar
- Weston, J., Chapelle, O., Elisseeff, A., Schölkopf, B., & Vapnik, V. (2002). Kernel dependency estimation. In Advances in neural information processing systems 15 [Neural Information Processing Systems, NIPS 2002, December 9–14, 2002, Vancouver, British Columbia, Canada], pp. 873–880.Google Scholar
- Wold, H. (1985). Partial least squares. In S. Kotz & N. L. Johnson (Eds.), Encyclopedia of statistical sciences (Vol. 6, pp. 581–591). New York: Wiley.Google Scholar
- Wolpert, D. H. (1992). Stacked generalization. Neural Networks, 5(2), 241–259.MathSciNetCrossRefGoogle Scholar
- Wolpert, D. H. (1996). The lack of a priori distinctions between learning algorithms. Neural Computation, 8(7), 1341–1390.CrossRefGoogle Scholar
- Wolpert, D. H. (2002). The supervised learning no-free-lunch theorems. In R. Roy, M. Köppen, S. Ovaska, T. Furuhashi, & F. Hoffmann (Eds.), Soft computing and industry: Recent applications (pp. 25–42). London: Springer.Google Scholar
- Yeh, I. C. (2007). Modeling slump flow of concrete using second-order regressions and artificial neural networks. Cement and Concrete Composites, 29(6), 474–480.CrossRefGoogle Scholar
- Zhang, M., & Zhou, Z. (2014). A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8), 1819–1837.CrossRefGoogle Scholar
- Zhang, Y., & Schneider, J. G. (2011). Multi-label output codes using canonical correlation analysis. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, AISTATS 2011, Fort Lauderdale, USA, April 11–13, 2011, pp. 873–882.Google Scholar
- Zhang, Y., & Schneider, J. G. (2012). Maximum margin output coding. In Proceedings of the 29th international conference on machine learning, ICML 2012, Edinburgh, Scotland, UK, June 26–July 1, 2012.Google Scholar
- Zhou, J., Chen, J., & Ye, J. (2011a). Clustered multi-task learning via alternating structure optimization. In Advances in neural information processing systems 24: 25th annual conference on neural information processing systems 2011. Proceedings of a meeting held 12–14 December 2011, Granada, Spain, pp. 702–710.Google Scholar
- Zhou, J., Chen, J., & Ye, J. (2011b). Malsar: Multi-task learning via structural regularization. Tempe: Arizona State University.Google Scholar
- Zhou, J., Chen, J., & Ye, J. (2012). Multi-task learning: Theory, algorithms, and applications. https://www.siam.org/meetings/sdm12/zhou_chen_ye.pdf