Advertisement

A Replicator Dynamics Approach to Collective Feature Engineering in Random Forests

  • Khaled Fawgreh
  • Mohamed Medhat Gaber
  • Eyad Elyan
Conference paper

Abstract

It has been demonstrated how random subspaces can be used to create a Diversified Random Forest, which in turn can lead to better performance in terms of predictive accuracy. Motivated by the fact that each subsforest is built using a set of features that can overlap with those sets of features in other subforests, we hypothesise that using Replicator Dynamics can perform a collective feature engineering, by allowing subforests with better performance to grow and those with lower performance to shrink. In this paper, we propose a new method to further improve the performance of Diversified Random Forest using Replicator Dynamics which has been used extensively in evolutionary game dynamics. A thorough experimental study on 15 real datasets showed favourable results, demonstrating the potential of the proposed method. Some experiments reported a boost in predictive accuracy of over 10 % consistently, evidencing the effectiveness of the iterative feature engineering achieved through the Replicator Dynamics procedure.

Keywords

Feature Engineering Replicator Dynamics Concept Drift Ensemble Learning Evolutionary Game Theory 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Adeva, J.J.G., Beresi, U., Calvo, R.: Accuracy and diversity in ensembles of text categorisers. CLEI Electron. J. 9(1) (2005)Google Scholar
  2. 2.
    Bache, K., Lichman, M.: Uci machine learning repository (2013)Google Scholar
  3. 3.
    Bomze, I.M.: Lotka-volterra equation and replicator dynamics: new issues in classification. Biol. Cybern. 72(5), 447–453 (1995)CrossRefMATHGoogle Scholar
  4. 4.
    Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)MATHGoogle Scholar
  5. 5.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefMATHGoogle Scholar
  6. 6.
    Brown, G., Wyatt, J., Harris, R., Yao, X.: Diversity creation methods: a survey and categorisation. Inf. Fusion 6(1), 5–20 (2005)CrossRefGoogle Scholar
  7. 7.
    Cuzzocrea, A., Francis, S.L., Gaber, M.M.: An information-theoretic approach for setting the optimal number of decision trees in random forests. In: Systems, Man, and Cybernetics (SMC), 2013 IEEE International Conference on, pp. 1013–1019. IEEE (2013)Google Scholar
  8. 8.
    Fawagreh, K., Gaber, M.M., Elyan, E.: Diversified random forests using random subspaces. In: Intelligent Data Engineering and Automated Learning–IDEAL 2014, pp. 85–92. Springer (2014)Google Scholar
  9. 9.
    Fawagreh, K., Gaber, M.M., Elyan, E.: Random forests: from early developments to recent advancements. Syst. Sci. Control Eng: Open Access J. |bf 2(1), 602–609 (2014)Google Scholar
  10. 10.
    Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D.: Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res. 15, 3133–3181 (2014)MathSciNetMATHGoogle Scholar
  11. 11.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55(1), 119–139 (1997)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Galstyan, A.: Continuous strategy replicator dynamics for multi-agent Q-learning. Auton. Agents Multi-agent Syst. 26(1), 37–53 (2013)CrossRefGoogle Scholar
  13. 13.
    Hauert, C.: Replicator dynamics of reward & reputation in public goods games. J. Theor. Biol. 267(1), 22–28 (2010)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Hauert, C., De Monte, S., Hofbauer, J., Sigmund, K.: Replicator dynamics for optional public good games. J. Theor. Biol. 218(2), 187–194 (2002)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Hilbe, C.: Local replicator dynamics: a simple link between deterministic and stochastic models of evolutionary game theory. Bull. Math. Biol. 73(9), 2068–2087 (2011)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Hofbauer, J., Sigmund, K.: Evolutionary game dynamics. Bull. Am. Math. Soc. 40(4), 479–519 (2003)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    Hutson, V., Schmitt, K.: Permanence and the dynamics of biological systems. Math. Biosci. 111(1), 1–71 (1992)MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach. Learn. 51(2), 181–207 (2003)CrossRefMATHGoogle Scholar
  19. 19.
    Lohmann, G., Bohn, S.: Using replicator dynamics for analyzing fMRI data of the human brain. IEEE Trans. Med. Imag. 21(5), 485–492 (2002)CrossRefGoogle Scholar
  20. 20.
    Maclin, R., Opitz, D.: Popular ensemble methods: an empirical study. arXiv:1106.0257 (2011) (preprint)
  21. 21.
    Nowak, M.A., Sigmund, K.: Evolutionary dynamics of biological games. Science 303(5659), 793–799 (2004)CrossRefGoogle Scholar
  22. 22.
    Olfati-Saber, R.: Evolutionary dynamics of behavior in social networks. In: Decision and Control, 2007 46th IEEE Conference on, pp. 4051–4056. IEEE (2007)Google Scholar
  23. 23.
    Polikar, R.: Ensemble based systems in decision making. Circuits Syst. Mag. IEEE 6(3), 21–45 (2006)CrossRefGoogle Scholar
  24. 24.
    Roca, C.P., Cuesta, J.A., Sánchez, A.: Evolutionary game theory: temporal and spatial effects beyond replicator dynamics. Phys. Life Rev. 6(4), 208–249 (2009)CrossRefGoogle Scholar
  25. 25.
    Rokach, L.: Ensemble-based classifiers. Artif. Intell. Rev. 33(1–2), 1–39 (2010)CrossRefGoogle Scholar
  26. 26.
    Tang, E.K., Suganthan, P.N., Yao, X.: An analysis of diversity measures. Mach. Learn. 65(1), 247–271 (2006)CrossRefGoogle Scholar
  27. 27.
    Taylor, P.D., Jonker, L.B.: Evolutionary stable strategies and game dynamics. Math. Biosci. 40(1), 145–156 (1978)MathSciNetCrossRefMATHGoogle Scholar
  28. 28.
    Wolpert, D.H.: Stacked generalization. Neural Netw. 5(2), 241–259 (1992)CrossRefGoogle Scholar
  29. 29.
    Yan, W., Goebel, K.F.: Designing classifier ensembles with constrained performance requirements. In: Defense and Security, pp. 59–68. International Society for Optics and Photonics (2004)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Khaled Fawgreh
    • 1
  • Mohamed Medhat Gaber
    • 1
  • Eyad Elyan
    • 1
  1. 1.School of Computing Science and Digital MedialRobert Gordon UniversityAberdeenUK

Personalised recommendations