Advertisement

Journal of Signal Processing Systems

, Volume 91, Issue 2, pp 191–215 | Cite as

Multi-Output Tree Chaining: An Interpretative Modelling and Lightweight Multi-Target Approach

  • Saulo Martiello MasteliniEmail author
  • Victor Guilherme Turrisi da Costa
  • Everton Jose Santana
  • Felipe Kenji Nakano
  • Rodrigo Capobianco Guido
  • Ricardo Cerri
  • Sylvio BarbonJr.
Article

Abstract

Multi-target regression (MTR) regards predictive problems with multiple numerical targets. To solve this, machine learning techniques can model solutions treating each target as a separated problem based only on the input features. Nonetheless, modelling inter-target correlation can improve predictive performance. When performing MTR tasks using the statistical dependencies of targets, several approaches put aside the evaluation of each pair-wise correlation between those targets, which may differ for each problem. Besides that, one of the main drawbacks of the current leading MTR method is its high memory cost. In this paper, we propose a novel MTR method called Multi-output Tree Chaining (MOTC) to overcome the mentioned disadvantages. Our method provides an interpretative internal tree-based structure which represents the relationships between targets denominated Chaining Trees (CT). Different from the current techniques, we compute the outputs dependencies, one-by-one, based on the Random Forest importance metric. Furthermore, we proposed a memory friendly approach which reduces the number of required regression models when compared to a leading method, reducing computational cost. We compared the proposed algorithm against three MTR methods (Single-target - ST; Multi-Target Regressor Stacking - MTRS; and Ensemble of Regressor Chains - ERC) on 18 benchmark datasets with two base regression algorithms (Random Forest and Support Vector Regression). The obtained results show that our method is superior to the ST approach regarding predictive performance, whereas, having no significant difference from ERC and MTRS. Moreover, the interpretative tree-based structures built by MOTC pose as great insight on the relationships among targets. Lastly, the proposed solution used significantly less memory than ERC being very similar in predictive performance.

Keywords

Multi-target regression Multi-output Memory-friendly algorithm Interpretative tree structure Machine learning 

Notes

Acknowledgements

The authors would like to thank CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior), CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico) and FAPESP (Fundação de Amparo à Pesquisa do Estado de São Paulo) for financial support.

References

  1. 1.
    Aho, T., Zenko, B., Dzeroski, S., Elomaa, T. (2012). Multi-target regression with rule ensembles. Journal of Machine Learning Research, 13, 2367–2407.MathSciNetzbMATHGoogle Scholar
  2. 2.
    Borchani, H., Varando, G., Bielza, C., Larrañaga, P. (2015). A survey on multi-output regression. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5(5), 216–233.Google Scholar
  3. 3.
    Breiman, L. (2001). Random forests. Machine learning, 45.1, 5–32.  https://doi.org/10.1017/CBO9781107415324.004.CrossRefzbMATHGoogle Scholar
  4. 4.
    Brugger, D., Rosenstiel, W., Bogdan, M. (2011). Online SVR training by solving the primal optimization problem. Journal of Signal Processing Systems, 65(3), 391–402.CrossRefGoogle Scholar
  5. 5.
    Chen, H., & Ser, W. (2011). Sound source DOA estimation and localization in noisy reverberant environments using least-squares support vector machines. Journal of Signal Processing Systems, 63(3), 287–300.CrossRefGoogle Scholar
  6. 6.
    Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297.  https://doi.org/10.1023/A:1022627411411.CrossRefzbMATHGoogle Scholar
  7. 7.
    Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7, 1–30.MathSciNetzbMATHGoogle Scholar
  8. 8.
    Di Persio, L., & Honchar, O. (2016). Artificial neural networks architectures for stock price prediction: comparisons and applications. International Journal of Circuits, Systems and Signal Processing, 10, 403–413.Google Scholar
  9. 9.
    Drucker, H., Burges, C.J.C., Kaufman, L., Smola, A.J., Vapnik, V. (1997). Support vector regression machines. In Mozer, M.C., Jordan, M.I., Petsche, T. (Eds.) Advances in neural information processing systems (Vol. 9, pp. 155–161). MIT Press. http://papers.nips.cc/paper/1238-support-vector-regression-machines.pdf.
  10. 10.
    Evgeniou, T., Figueiras-Vidal, A.R., Theodoridis, S. (2008). Emerging machine learning techniques in signal processing.Google Scholar
  11. 11.
    Gama, J., & Brazdil, P. (2000). Cascade generalization. Machine Learning, 41(3), 315–343.  https://doi.org/10.1023/A:1007652114878.CrossRefzbMATHGoogle Scholar
  12. 12.
    Genuer, R., Poggi, J.M., Tuleau-Malot, C. (2010). Variable selection using random forests. Pattern Recognition Letters, 31(14), 2225–2236.  https://doi.org/10.1016/j.patrec.2010.03.014. http://www.sciencedirect.com/science/article/pii/S0167865510000954.CrossRefGoogle Scholar
  13. 13.
    Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301), 13–30.  https://doi.org/10.1080/01621459.1963.10500830. http://amstat.tandfonline.com/doi/abs/10.1080/01621459.1963.10500830.MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Katagiri, S., Nakamura, A., Adali, T., Tao, J., Larsen, J., Tan, T. (2014). Guest editorial: Machine learning for signal processing. Journal of Signal Processing Systems, 74(3), 281–283.  https://doi.org/10.1007/s11265-014-0871-6.CrossRefGoogle Scholar
  15. 15.
    Kocev, D., Džeroski, S., White, M.D., Newell, G.R., Griffioen, P. (2009). Using single- and multi-target regression trees and ensembles to model a compound index of vegetation condition. Ecological Modelling, 220(8), 1159–1168.CrossRefGoogle Scholar
  16. 16.
    Kocev, D., Vens, C., Struyf, J., Džeroski, S. (2007). Ensembles of multi-objective decision trees. In European conference on machine learning (pp. 624–631). Springer.Google Scholar
  17. 17.
    Kocev, D., Vens, C., Struyf, J., Džeroski, S. (2013). Tree ensembles for predicting structured outputs. Pattern Recognition, 46(3), 817–833.CrossRefGoogle Scholar
  18. 18.
    Li, X., & Zheng, J. (2016). Active learning for regression with correlation matching and labeling error suppression. IEEE Signal Processing Letters, 23(8), 1081–1085.CrossRefGoogle Scholar
  19. 19.
    Lichman, M. (2013). UCI machine learning repository. http://archive.ics.uci.edu/ml.
  20. 20.
    Mastelini, S.M., Santana, E.J., Cerri, R., Barbon, S. Jr. (2017). DSTARS: a multi-target deep structure for tracking asynchronous regressor stack. In Brazilian conference on intelligent systems. BRACIS 2017.Google Scholar
  21. 21.
    Melki, G., Cano, A., Kecman, V., Ventura, S. (2017). Multi-target support vector regression via correlation regressor chains. Information Sciences, 415, 53–69.MathSciNetCrossRefGoogle Scholar
  22. 22.
    Moyano, J.M., Gibaja, E.L., Ventura, S. (2017). An evolutionary algorithm for optimizing the target ordering in ensemble of regressor chains. In 2017 IEEE congress on evolutionary computation (CEC) (pp. 2015–2021). IEEE.Google Scholar
  23. 23.
    Santana, E.J., Mastelini, S.M., Barbon, S. Jr. (2017). Deep regressor stacking for air ticket prices prediction. In Brazilian symposium of information systems (pp. 216–233). SBSI 2017.Google Scholar
  24. 24.
    Sidike, P., Krieger, E., Alom, M.Z., Asari, V.K., Taha, T. (2017). A fast single-image super-resolution via directional edge-guided regularized extreme learning regression. In Signal, image and video processing (pp. 1–8).Google Scholar
  25. 25.
    Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W., Vlahavas, I. (2016). Multi-target regression via input space expansion: treating targets as inputs. Machine Learning, 104(1), 55–98.MathSciNetCrossRefGoogle Scholar
  26. 26.
    Tsoumakas, G., Spyromitros-Xioufis, E., Vrekou, A., Vlahavas, I. (2014). Multi-target regression via random linear target combinations. In Joint european conference on machine learning and knowledge discovery in databases (pp. 225–240). Springer.Google Scholar
  27. 27.
    Wang, Q., Wu, Y., Shen, Y., Liu, Y., Lei, Y. (2015). Supervised sparse manifold regression for head pose estimation in 3d space. Signal Processing, 112, 34–42.CrossRefGoogle Scholar
  28. 28.
    Watanabe, S., Nakamura, A., Juang, B.H.F. (2014). Structural bayesian linear regression for hidden Markov models. Journal of Signal Processing Systems, 74(3), 341–358.CrossRefGoogle Scholar
  29. 29.
    Zhang, W., Liu, X., Ding, Y., Shi, D. (2012). Multi-output LS-SVR machine in extended feature space. In CIMSA 2012 - 2012 IEEE Int. Conf. Comput. Int.ll. Meas. Syst. Appl. Proc. (pp. 130–144).  https://doi.org/10.1109/CIMSA.2012.6269600.

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Saulo Martiello Mastelini
    • 1
    Email author
  • Victor Guilherme Turrisi da Costa
    • 1
  • Everton Jose Santana
    • 2
  • Felipe Kenji Nakano
    • 3
  • Rodrigo Capobianco Guido
    • 4
  • Ricardo Cerri
    • 3
  • Sylvio BarbonJr.
    • 1
  1. 1.Computer Science DepartmentState University of Londrina. Rodovia Celso Garcia Cid, Km 380, s/n - Campus UniversitárioLondrinaBrazil
  2. 2.Electrical Engineering DepartmentState University of Londrina. Rodovia Celso Garcia Cid, Km 380, s/n - Campus UniversitárioLondrinaBrazil
  3. 3.Department of Computer ScienceFederal University of São Carlos, Rodovia Washington LuísSão CarlosBrazil
  4. 4.Instituto de Biociências, Letras e Ciências ExatasUnesp - Univ Estadual Paulista (São Paulo State University)São José do Rio PretoBrazil

Personalised recommendations