Soft Computing

, Volume 21, Issue 4, pp 1065–1080 | Cite as

Ensemble bayesian networks evolved with speciation for high-performance prediction in data mining

  • Kyung-Joong Kim
  • Sung-Bae Cho
Methodologies and Application


Bayesian networks (BNs) can be easily refined (or learn) using data given prior knowledge about a changing environment. Furthermore, by exploring multiple diverse BNs in parallel, it is expected that an intelligent system may adapt quickly to changes in the environment, resulting in robust prediction. Recently, there have been attempts to design BN structures using evolutionary algorithms; however, most of these have used only the fittest solution from the final generation. Because it is difficult to combine all of the important factors into a single evaluation function, the solution is often biased and of limited adaptability. Here we describe a method of generating diverse BN structures via speciation and selective combination for adaptive prediction. Experiments using the seven benchmark networks show that the proposed method can result in improved accuracy in handling uncertainty by exploiting ensembles of BNs evolved by speciation.


Prediction Bayesian networks Uncertainty Ensemble Speciation Evolution 



This work was supported by the National Research Foundation of Korea (NRF) Grant funded by the Korean Government (MSIP) (2013 R1A2A2A01016589) and the Industrial Strategic Technology Development Program, 10044828, Development of augmenting multisensory technology for enhancing significant effect on service industry, funded by the Ministry of Trade, Industry & Energy (MI, Korea).


  1. Barber D (2012) Bayesian reasoning and machine learning. Cambridge University Press, CambridgezbMATHGoogle Scholar
  2. Beinlich IA, Suermondt HJ, Chavez RM, Cooper GF (1989) The ALARM monitoring system: a case study with two probabilistic inference techniques for belief networks. In: Proceedings of the Second European Conference on Artificial Intelligence in Medicine, pp 247–256Google Scholar
  3. Binder J, Koller D, Russell S, Kanazawa K (1997) Adaptive probabilistic networks with hidden variables. Mach Learn 29(2–3):213–244CrossRefzbMATHGoogle Scholar
  4. Chickering DM, Geiger D, Heckerman D (1994) Learning Bayesian networks is NP-hard, Technical Report MSR-TR-94-17, Microsoft ResearchGoogle Scholar
  5. Colace F, De Santo M, Greco L (2014) Learning Bayesian network structure using a multiexpert approach. Int J Softw Eng Knowl Eng 24(2):269–284CrossRefGoogle Scholar
  6. Cooper GF, Herskovits E (1992) A Bayesian method for the induction of probabilistic networks from data. Mach Learn 9:309–347zbMATHGoogle Scholar
  7. Daly R, Shen Q, Aitken S (2011) Learning Bayesian networks: approaches and issues. Knowl Eng Rev 26(2):99–157CrossRefGoogle Scholar
  8. Feng G, Zhang J-D, Liao SS (2014) A novel method for combining Bayesian networks, theoretical analysis, and its applications. Pattern Recognit 47(5):2057–2069CrossRefzbMATHGoogle Scholar
  9. Gamez JA, Mateo JL, Puerta JM (2011) Learning Bayesian networks by hill climbing: efficient methods based on progressive restriction of the neighborhood. Data Min Knowl Discov 22(1–2):106–148MathSciNetCrossRefzbMATHGoogle Scholar
  10. Garg A, Pavlovic V, Rehg JM (2003) Boo sted learning in dynamic Bayesian networks for multimodal speaker detection. Proc IEEE 91(9):1355–1369CrossRefGoogle Scholar
  11. Goldberg DE (2008) Genetic algorithms in search, optimization, and machine learning, 1st edn. Addison-Wesley ProfessionalGoogle Scholar
  12. Gouvea MM Jr., Araujo AFR (2010) Diversity-based adaptive evolutionary algorithms, Chapter 1. New Achievements in Evolutionary ComputationGoogle Scholar
  13. Heckerman D (2008) A tutorial on learning with Bayesian networks. Innov Bayesian Netw 156:33–82CrossRefzbMATHGoogle Scholar
  14. Hu L, Wang L (2013) Using consensus Bayesian network to model the reactive oxygen species regulatory pathway. PLOS One 8(2):e56832. doi: 10.1371/journal.pone.0056832
  15. Hwang K-S, Cho S-B (2009) Landmark detection from mobile life log using a modular Bayesian network model. Expert Syst Appl 36:12065–12076CrossRefGoogle Scholar
  16. Jensen FV, Kjærulff U, Olesen KG, Pedersen J (1989) An expert system for control of waste water treatment—a pilot project. Technical report. Judex Datasystemer A/S, Aalborg (in Danish)Google Scholar
  17. Kim K-J, Cho S-B (2005) Systematically incorporating domain-specific knowledge into evolutionary speciated checkers players. IEEE Trans Evol Comput 9(6):615–627CrossRefGoogle Scholar
  18. Kim K-J, Cho S-B (2008) Evolutionary ensemble of diverse artificial neural networks using speciation. Neurocomputing 71(7–9):1604–1618CrossRefGoogle Scholar
  19. Kim K-J, Cho S-B (2012) Automated synthesis of multiple analog circuits using evolutionary computation for redundancy-based fault-tolerance. Appl Soft Comput 12(4):1309–1321CrossRefGoogle Scholar
  20. Kim K, Mckay R (2012) Stochastic diversity loss and scalability in estimation of distribution genetic programming. IEEE Trans Evol Comput 17(3):301–320Google Scholar
  21. Kim K-J, Park J-G, Cho S-B (2011) Correlation analysis and performance evaluation of distance measures for evolutionary neural networks. J Intell Fuzzy Syst 22:83–92MathSciNetzbMATHGoogle Scholar
  22. Kim KJ, Yoo JO, Cho SB (2005) Robust inference of Bayesian networks using speciated evolution and ensemble. In: International Symposium on Methodologies for Intelligent Systems, pp 92–101Google Scholar
  23. Koller D, Friedman N (2009) Probabilistic graphical models: principles and techniques. MIT Press, CambridgezbMATHGoogle Scholar
  24. Korb KB, Nicholson AE (2010) Bayesian artificial intelligence, 2nd edn. CRC Press, Boca RatonGoogle Scholar
  25. Larranaga P, Karshenas H, Bielza C, Santana R (2013) A review on evolutionary algorithms in Bayesian network learning and inference tasks. Inf Sci 233(1):109–125MathSciNetCrossRefzbMATHGoogle Scholar
  26. Larranaga P, Kuijpers CMH, Murga RH, Yurramendi Y (1996) Learning Bayesian network structures by searching for the best ordering with genetic algorithm. IEEE Trans Syst Man Cybern Part A 26(4):487–493Google Scholar
  27. Larranaga P, Poza M, Yurramendi Y, Murga RH, Kuijpers CMH (1996) Structure learning of Bayesian networks by genetic algorithms: a performance analysis of control parameters. IEEE Trans Pattern Anal Mach Intell 18(9):912–926Google Scholar
  28. Lauritzen S-L, Spiegelhalter DJ (1988) Local computations with probabilities on graphical structures and their applications on expert systems. J R Stat Soc B 50(2):157–224MathSciNetzbMATHGoogle Scholar
  29. Li XL, He XD, Yuan SM (2005) Learning Bayesian networks structures from incomplete data based on extending evolutionary programming. In: Proceedings of the Fourth International Conference on Machine Learning and Cybernetics, pp 2039–2043Google Scholar
  30. Li W, Liu W, Yue K (2008) Recovering the global structure from multiple local Bayesian networks. Int J Artif Intell Tools 17(6):1067–1088CrossRefGoogle Scholar
  31. Luo X, Ouyang Y, Xiong Z (2011) Improving matrix factorization-based recommender via ensemble methods. Int J Inf Technol Decision Making 10(3):539–561CrossRefGoogle Scholar
  32. Mahfoud SW (1995) Niching methods for genetic algorithms. Ph.D. Dissertation, University of Illinois at Urbana-ChampaignGoogle Scholar
  33. Muruzabal J, Cotta C (2007) A Study on the evolution of Bayesian network graph structures. Adv Probab Graph Models 193–214Google Scholar
  34. Myers JW, Laskey KB, Dejong KA (1999) Learning Bayesian networks from incomplete data using evolutionary algorithm. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp 458–465Google Scholar
  35. Na Y, Yang J (2010) Distributed Bayesian network structure learning. In: IEEE International Symposium on Industrial Electronics, pp 1607–161Google Scholar
  36. Pena JM, Kocka T, Nielsen JD (2004) Featuring multiple local optima to assist the user in the interpretation of induced Bayesian network models. In: Proceedings of the Tenth International Conference on Information Processing and Management of Uncertainty in Knowledge-based Systems, pp 1683–1690Google Scholar
  37. Peng Y, Kou G, Wang G, Wu W, Shi Y (2011) Ensemble of software defect predictors: an AHP-based evaluation method. Int J Inf Technol Decision Making 10(1):187–206CrossRefGoogle Scholar
  38. Robles V, Larranaga P, Pena JM, Menasalvas E, Perez MS, Herves V, Wasilewska A (2004) Bayesian network multi-classifiers for protein secondary structure prediction. Artif Intell Med 31(2):117–136CrossRefGoogle Scholar
  39. Rogers A, Prugel-Bennett A (1999) Genetic drift in genetic algorithm selection schemes. IEEE Trans Evol Comput 3(4):298–303CrossRefGoogle Scholar
  40. Schadt EE, Linderman MD, Sorenson J, Lee L, Nolan GP (2010) Computational solutions to large-scale data management and analysis. Nat Rev Genet 11(9):647–657CrossRefGoogle Scholar
  41. Scutari M, Denis JB (2014) Bayesian networks: with examples in R. Chapman & Hall, LondonzbMATHGoogle Scholar
  42. Shen C-W (2009) A Bayesian networks approach to modeling financial risks of e-logistics investments. Int J Inf Technol Decision Making 8(4):711–726CrossRefzbMATHGoogle Scholar
  43. Su X, Khoshgoftaar TM (2008) Collaborative filtering for multi-class data using Bayesian networks. Int J Artif Intell Tools 17(1):71–85CrossRefGoogle Scholar
  44. Vafaee F (2014) Learning the structure of large-scale Bayesian networks using genetic algorithm. In: Proceedings of the Conference on Genetic and Evolutionary Computation, pp 855–862Google Scholar
  45. Wong ML, Lam W, Leung KS (1999) Using evolutionary programming and minimum description length principle for data mining of Bayesian networks. IEEE Trans Pattern Anal Mach Intell 21(2):174–178Google Scholar
  46. Wong ML, Lee SY, Leung KS (2004) Data mining of Bayesian networks using cooperative coevolution. Decision Support Syst 38:451–472CrossRefGoogle Scholar
  47. Zhou ZH (2012) Ensemble methods: foundations and algorithms. Chapman & Hall/CRC, LondonGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringSejong UniversitySeoulSouth Korea
  2. 2.Department of Computer ScienceYonsei UniversitySeoulSouth Korea

Personalised recommendations