Multi-Output Tree Chaining: An Interpretative Modelling and Lightweight Multi-Target Approach
Multi-target regression (MTR) regards predictive problems with multiple numerical targets. To solve this, machine learning techniques can model solutions treating each target as a separated problem based only on the input features. Nonetheless, modelling inter-target correlation can improve predictive performance. When performing MTR tasks using the statistical dependencies of targets, several approaches put aside the evaluation of each pair-wise correlation between those targets, which may differ for each problem. Besides that, one of the main drawbacks of the current leading MTR method is its high memory cost. In this paper, we propose a novel MTR method called Multi-output Tree Chaining (MOTC) to overcome the mentioned disadvantages. Our method provides an interpretative internal tree-based structure which represents the relationships between targets denominated Chaining Trees (CT). Different from the current techniques, we compute the outputs dependencies, one-by-one, based on the Random Forest importance metric. Furthermore, we proposed a memory friendly approach which reduces the number of required regression models when compared to a leading method, reducing computational cost. We compared the proposed algorithm against three MTR methods (Single-target - ST; Multi-Target Regressor Stacking - MTRS; and Ensemble of Regressor Chains - ERC) on 18 benchmark datasets with two base regression algorithms (Random Forest and Support Vector Regression). The obtained results show that our method is superior to the ST approach regarding predictive performance, whereas, having no significant difference from ERC and MTRS. Moreover, the interpretative tree-based structures built by MOTC pose as great insight on the relationships among targets. Lastly, the proposed solution used significantly less memory than ERC being very similar in predictive performance.
KeywordsMulti-target regression Multi-output Memory-friendly algorithm Interpretative tree structure Machine learning
The authors would like to thank CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior), CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico) and FAPESP (Fundação de Amparo à Pesquisa do Estado de São Paulo) for financial support.
- 2.Borchani, H., Varando, G., Bielza, C., Larrañaga, P. (2015). A survey on multi-output regression. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5(5), 216–233.Google Scholar
- 8.Di Persio, L., & Honchar, O. (2016). Artificial neural networks architectures for stock price prediction: comparisons and applications. International Journal of Circuits, Systems and Signal Processing, 10, 403–413.Google Scholar
- 9.Drucker, H., Burges, C.J.C., Kaufman, L., Smola, A.J., Vapnik, V. (1997). Support vector regression machines. In Mozer, M.C., Jordan, M.I., Petsche, T. (Eds.) Advances in neural information processing systems (Vol. 9, pp. 155–161). MIT Press. http://papers.nips.cc/paper/1238-support-vector-regression-machines.pdf.
- 10.Evgeniou, T., Figueiras-Vidal, A.R., Theodoridis, S. (2008). Emerging machine learning techniques in signal processing.Google Scholar
- 12.Genuer, R., Poggi, J.M., Tuleau-Malot, C. (2010). Variable selection using random forests. Pattern Recognition Letters, 31(14), 2225–2236. https://doi.org/10.1016/j.patrec.2010.03.014. http://www.sciencedirect.com/science/article/pii/S0167865510000954.CrossRefGoogle Scholar
- 13.Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301), 13–30. https://doi.org/10.1080/01621459.1963.10500830. http://amstat.tandfonline.com/doi/abs/10.1080/01621459.1963.10500830.MathSciNetCrossRefzbMATHGoogle Scholar
- 16.Kocev, D., Vens, C., Struyf, J., Džeroski, S. (2007). Ensembles of multi-objective decision trees. In European conference on machine learning (pp. 624–631). Springer.Google Scholar
- 19.Lichman, M. (2013). UCI machine learning repository. http://archive.ics.uci.edu/ml.
- 20.Mastelini, S.M., Santana, E.J., Cerri, R., Barbon, S. Jr. (2017). DSTARS: a multi-target deep structure for tracking asynchronous regressor stack. In Brazilian conference on intelligent systems. BRACIS 2017.Google Scholar
- 22.Moyano, J.M., Gibaja, E.L., Ventura, S. (2017). An evolutionary algorithm for optimizing the target ordering in ensemble of regressor chains. In 2017 IEEE congress on evolutionary computation (CEC) (pp. 2015–2021). IEEE.Google Scholar
- 23.Santana, E.J., Mastelini, S.M., Barbon, S. Jr. (2017). Deep regressor stacking for air ticket prices prediction. In Brazilian symposium of information systems (pp. 216–233). SBSI 2017.Google Scholar
- 24.Sidike, P., Krieger, E., Alom, M.Z., Asari, V.K., Taha, T. (2017). A fast single-image super-resolution via directional edge-guided regularized extreme learning regression. In Signal, image and video processing (pp. 1–8).Google Scholar
- 26.Tsoumakas, G., Spyromitros-Xioufis, E., Vrekou, A., Vlahavas, I. (2014). Multi-target regression via random linear target combinations. In Joint european conference on machine learning and knowledge discovery in databases (pp. 225–240). Springer.Google Scholar
- 29.Zhang, W., Liu, X., Ding, Y., Shi, D. (2012). Multi-output LS-SVR machine in extended feature space. In CIMSA 2012 - 2012 IEEE Int. Conf. Comput. Int.ll. Meas. Syst. Appl. Proc. (pp. 130–144). https://doi.org/10.1109/CIMSA.2012.6269600.