Advertisement

Searching Parsimonious Solutions with GA-PARSIMONY and XGBoost in High-Dimensional Databases

  • Francisco Javier Martinez-de-PisonEmail author
  • Esteban Fraile-Garcia
  • Javier Ferreiro-Cabello
  • Rubén Gonzalez
  • Alpha Pernia
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 527)

Abstract

EXtreme Gradient Boosting (XGBoost) has become one of the most successful techniques in machine learning competitions. It is computationally efficient and scalable, it supports a wide variety of objective functions and it includes different mechanisms to avoid over-fitting and improve accuracy. Having so many tuning parameters, soft computing (SC) is an alternative to search precise and robust models against classical hyper-tuning methods. In this context, we present a preliminary study in which a SC methodology, named GA-PARSIMONY, is used to find accurate and parsimonious XGBoost solutions. The methodology was designed to optimize the search of parsimonious models by feature selection, parameter tuning and model selection. In this work, different experiments are conducted with four complexity metrics in six high dimensional datasets. Although XGBoost performs well with high-dimensional databases, preliminary results indicated that GA-PARSIMONY with feature selection slightly improved the testing error. Therefore, the choice of solutions with fewer inputs, between those with similar cross-validation errors, can help to obtain more robust solutions with better generalization capabilities.

Keywords

XGBoost Genetic algorithms Parameter tuning Parsimony criterion GA-PARSIMONY 

Notes

Acknowledgments

The authors would like to acknowledge the fellowship APPI15/05 granted by the Banco Santander and the University of La Rioja.

References

  1. 1.
    Ahila, R., Sadasivam, V., Manimala, K.: An integrated PSO for parameter determination and feature selection of ELM and its application in classification of power system disturbances. Appl. Soft Comput. 32, 23–37 (2015)CrossRefGoogle Scholar
  2. 2.
    Antonanzas-Torres, F., Urraca, R., Antonanzas, J., Fernandez-Ceniceros, J., de Pison, F.M.: Generation of daily global solar irradiation with support vector machines for regression. Energy Convers. Manage. 96, 277–286 (2015)CrossRefGoogle Scholar
  3. 3.
    Caamaño, P., Bellas, F., Becerra, J.A., Duro, R.J.: Evolutionary algorithm characterization in real parameter optimization problems. Appl. Soft Comput. 13(4), 1902–1921 (2013)CrossRefGoogle Scholar
  4. 4.
    Chen, N., Ribeiro, B., Vieira, A., Duarte, J., Neves, J.C.: A genetic algorithm-based approach to cost-sensitive bankruptcy prediction. Expert Syst. Appl. 38(10), 12939–12945 (2011)CrossRefGoogle Scholar
  5. 5.
    Chen, T., He, T., Benesty, M.: xgboost: Extreme Gradient Boosting (2015). https://github.com/dmlc/xgboost, rpackageversion 0.4-3
  6. 6.
    Corchado, E., Wozniak, M., Abraham, A., de Carvalho, A.C.P.L.F., Snásel, V.: Recent trends in intelligent data analysis. Neurocomputing 126, 1–2 (2014)CrossRefGoogle Scholar
  7. 7.
    Dhiman, R., Saini, J., Priyanka: Genetic algorithms tuned expert model for detection of epileptic seizures from EEG signatures. Appl. Soft Comput. 19, 8–17 (2014)CrossRefGoogle Scholar
  8. 8.
    Ding, S.: Spectral and wavelet-based feature selection with particle swarm optimization for hyperspectral classification. J. Softw. 6(7), 1248–1256 (2011)CrossRefGoogle Scholar
  9. 9.
    Fernandez-Ceniceros, J., Sanz-Garcia, A., Antonanzas-Torres, F., de Pison, F.M.: A numerical-informational approach for characterising the ductile behaviour of the t-stub component. part 2: parsimonious soft-computing-based metamodel. Eng. Struct. 82, 249–260 (2015)CrossRefGoogle Scholar
  10. 10.
    Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 29(5), 1189–1232 (2001)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Huang, H.L., Chang, F.L.: ESVM: evolutionary support vector machine for automatic feature selection and classification of microarray data. Biosystems 90(2), 516–528 (2007)CrossRefGoogle Scholar
  12. 12.
    Kaggle: The home of data science. https://www.kaggle.com/
  13. 13.
    KDD-CUP: Annual data mining and knowledge discovery competition organized by ACM. http://www.kdd.org/kdd-cup
  14. 14.
    Michalewicz, Z., Janikow, C.Z.: Handling constraints in genetic algorithms. In: ICGA, pp. 151–157 (1991)Google Scholar
  15. 15.
    Oduguwa, V., Tiwari, A., Roy, R.: Evolutionary computing in manufacturing industry: an overview of recent applications. Appl. Soft Comput. 5(3), 281–299 (2005)CrossRefGoogle Scholar
  16. 16.
    Core Team, R.: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2013)Google Scholar
  17. 17.
    Reif, M., Shafait, F., Dengel, A.: Meta-learning for evolutionary parameter optimization of classifiers. Mach. Learn. 87(3), 357–380 (2012)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Sanz-Garcia, A., Fernandez-Ceniceros, J., Antonanzas-Torres, F., Pernia-Espinoza, A., Martinez-de Pison, F.J.: GA-PARSIMONY: a GA-SVR approach with feature selection and parameter optimization to obtain parsimonious solutions for predicting temperature settings in a continuous annealing furnace. Appl. Soft Comput. 35, 13–28 (2015)CrossRefGoogle Scholar
  19. 19.
    Sanz-Garcia, A., Fernández-Ceniceros, J., Fernández-Martínez, R., Martínez-de-Pisón, F.J.: Methodology based on genetic optimisation to develop overall parsimony models for predicting temperature settings on annealing furnace. Ironmaking Steelmaking 41(2), 87–98 (2014)CrossRefGoogle Scholar
  20. 20.
    Sanz-García, A., Fernández-Ceniceros, J., Antoñanzas-Torres, F., Martínez-de Pisón, F.J.: Parsimonious support vector machines modelling for set points in industrial processes based on genetic algorithm optimization. In: Herrero, Á., et al. (eds.) International Joint Conference SOCO13-CISIS13-ICEUTE13. Advances in Intelligent Systems and Computing, vol. 239, pp. 1–10. Springer International Publishing, Heidelberg (2014)CrossRefGoogle Scholar
  21. 21.
    Seni, G., Elder, J.: Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions. Morgan and Claypool Publishers, Chicago (2010)Google Scholar
  22. 22.
    Shaffer, J.P.: Modified sequentially rejective multiple test procedures. J. Am. Stat. Assoc. 81(395), 826–831 (1986)CrossRefGoogle Scholar
  23. 23.
    Urraca, R., Sanz-Garcia, A., Fernandez-Ceniceros, J., Sodupe-Ortega, E., Martinez-de-Pison, F.J.: Improving hotel room demand forecasting with a hybrid GA-SVR methodology based on skewed data transformation, feature selection and parsimony tuning. In: Onieva, E., Santos, I., Osaba, E., Quintián, H., Corchado, E. (eds.) HAIS 2015. LNCS (LNAI), vol. 9121, pp. 632–643. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-19644-2_52CrossRefGoogle Scholar
  24. 24.
    Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bull. 1(6), 80–83 (1945). http://dx.doi.org/10.2307/3001968CrossRefGoogle Scholar
  25. 25.
    Winkler, S.M., Affenzeller, M., Kronberger, G., Kommenda, M., Wagner, S., Jacak, W., Stekel, H.: Analysis of selected evolutionary algorithms in feature selection and parameter optimization for data based tumor marker modeling. In: Moreno-Díaz, R., Pichler, F., Quesada-Arencibia, A. (eds.) EUROCAST 2011. LNCS, vol. 6927, pp. 335–342. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-27549-4_43CrossRefGoogle Scholar
  26. 26.
    Xue, B., Zhang, M., Browne, W.N.: Particle swarm optimisation for feature selection in classification: novel initialisation and updating mechanisms. Appl. Soft Comput. 18, 261–276 (2014)CrossRefGoogle Scholar
  27. 27.
    Ye, J.: On measuring and correcting the effects of data mining and model selection. J. Am. Stat. Assoc. 93(441), 120–131 (1998)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (http://creativecommons.org/licenses/by-nc/2.5/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Authors and Affiliations

  • Francisco Javier Martinez-de-Pison
    • 1
    Email author
  • Esteban Fraile-Garcia
    • 1
  • Javier Ferreiro-Cabello
    • 1
  • Rubén Gonzalez
    • 1
  • Alpha Pernia
    • 1
  1. 1.EDMANS GroupUniversity of La RiojaLogroñoSpain

Personalised recommendations