Soft Computing

, Volume 15, Issue 10, pp 1881–1895 | Cite as

Structural learning of Bayesian networks using local algorithms based on the space of orderings

  • Juan I. Alonso-Barba
  • Luis delaOssa
  • Jose M. Puerta


Structural learning of Bayesian networks (BNs) is an NP-hard problem which is generally addressed by means of heuristic search algorithms. Despite the fact that earlier proposals for dealing with this task were based on searching the space of Directed Acyclic Graphs (DAGs), there are some alternative approaches. One of these approaches for structural learning consists of searching the space of orderings, as given a certain topological order among the problem variables, it is relatively easy to build (and evaluate) a BN compatible with it. In practice, the latter methods make it possible to obtain good results, but they are still costly in terms of computation. In this article, we prove the correctness of the method used to evaluate each ordering, and we propose some efficient learning algorithms based on it. Our first proposal is based on the Hill-Climbing algorithm, and uses an improved neighbourhood definition. The second algorithm is an extension of the first one, and is based on the well-known Variable Neighbourhood Search metaheuristic. Finally, iterative versions of both algorithms are also proposed. The algorithms have been tested over a set of different domains, and have been compared with other methods such as Hill-Climbing in the space of DAGs or Greedy Equivalent Search, in order to study their behaviour in practice.


Bayesian networks Structural learning Orderings Variable Neighbourhood Search 


  1. Andreassen S, Jensen FV, Andersen SK, Falck B, Kjrulff U, Woldbye M, Srensen AR, Rosenfalck A, Jensen F (1989) MUNIN—an expert EMG assistant, computer-aided electromyography and expert systems, chap 21Google Scholar
  2. Beinlich IA, Suermondt HJ, Chavez RM, Cooper GF (1989) The alarm monitoring system: a case study with two probabilistic inference techniques for belief networks. In: Second European Conference on Artificial Intelligence in Medicine, vol 38. Springer, Berlin, pp 247–256Google Scholar
  3. Binder J, Koller D, Russell SJ, Kanazawa K (1997) Adaptive probabilistic networks with hidden variables. Mach Learn 29(2–3):213–244zbMATHCrossRefGoogle Scholar
  4. Blanco R, Inza I, Larrañaga P (2003) Learning Bayesian networks in the space of structures by estimation of distribution algorithms. Int J Intell Syst 18(2):205–220zbMATHCrossRefGoogle Scholar
  5. Buntine W (1991) Theory refinement on Bayesian networks. In: Proceedings of the seventh conference (1991) on uncertainty in artificial intelligence (UAI’91). Morgan Kaufmann, Los Angeles, pp 52–60Google Scholar
  6. de Campos L, Puerta J (2001) Stochastic local algorithms for learning belief networks: searching in the space of the orderings. In: Symbolic and quantitative approaches to reasoning with uncertainty. Lecture notes in artificial intelligence, vol 2143. Springer, Berlin, pp 228–239Google Scholar
  7. Chickering D (2002) Optimal structure identification with greedy search. J Mach Learn Res 3:507–554MathSciNetGoogle Scholar
  8. Chickering D, Geiger D, Heckerman D (1995) Learning Bayesian networks: search methods and experimental results. In: Proceedings of the fifth conference on artificial intelligence and statistics, pp 112–128Google Scholar
  9. Chickering DM (1996) Learning Bayesian networks is NP-Complete. In: Fisher D, Lenz HJ (eds) Learning from data: artificial intelligence and statistics V. Springer, Berlin, pp 121–130Google Scholar
  10. Cooper G, Herskovits E (1992) A Bayesian method for the induction of probabilistic networks from data. Mach Learn 9:309–347zbMATHGoogle Scholar
  11. Cowell RG, Dawid AP, Lauritzen S, Spiegelhalter D (2003) Probabilistic networks and expert systems (information science and statistics). Springer, BerlinGoogle Scholar
  12. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MathSciNetGoogle Scholar
  13. Friedman M (1940) A comparison of alternative tests of significance for the problem of m rankings. Ann Math Stat 11(1):86–92zbMATHCrossRefGoogle Scholar
  14. Friedman N, Koller D (2003) Being Bayesian about network structure: a Bayesian approach to structure discovery in Bayesian networks. Mach Learn 50:95–126zbMATHCrossRefGoogle Scholar
  15. Gámez JA, Mateo JL, Puerta JM (2010) Learning Bayesian networks by hill climbing: efficient methods based on progressive restriction of the neighborhood. Data Min Knowl Discov (to appear). doi: 10.1007/s10618-010-0178-6
  16. García S, Herrera F (2008) An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons. J Mach Learn Res 9:2677–2694Google Scholar
  17. Heckerman D, Geiger D, Chickering D (1995) Learning Bayesian networks: the combination of knowledge and statistical data. Mach Learn 20(3):197–243zbMATHGoogle Scholar
  18. Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6:65–70MathSciNetzbMATHGoogle Scholar
  19. Jensen A, Jensen F (1996) Midas—an influece diagram for management of mildew in winter wheat. In: Proceedings of the 12th annual conference on uncertainty in artificial intelligence (UAI-96), pp 349–356Google Scholar
  20. Jensen CS (1997) Blocking Gibbs sampling for inference in large and complex Bayesian networks with applications in genetics. PhD thesis, Aalborg University, DenmarkGoogle Scholar
  21. Jensen F, Nielsen T (2007) Bayesian networks and decision graphs. Springer, BerlinGoogle Scholar
  22. Kristensen K, Rasmussen IA (2002) The use of a Bayesian network in the design of a decision support system for growing malting barley without use of pesticides. Comput Electron Agric 33:197–217CrossRefGoogle Scholar
  23. Larrañaga P, Poza M, Yurramendi Y, Murga R, Kuijpers C (1996) Structure learning of Bayesian networks by genetic algorithms: a performance analysis of control parameters. IEEE Trans Pattern Anal Mach Intell 18(9):912–926CrossRefGoogle Scholar
  24. Mladenović N, Hansen P (1997) Variable neighborhood search. Comput Oper Res 24:1097–1100. Google Scholar
  25. Pearl J (1988) Probabilistic reasoning in intelligent systems. Morgan Kaufmann, San MateoGoogle Scholar
  26. Teyssier M, Koller D (2005) Ordering-based search: a simple and effective algorithm for learning Bayesian networks. In: UAI ’05. Proceedings of the 21st conference in uncertainty in artificial intelligence. AUAI Press, Edinburgh, pp 548–549Google Scholar
  27. Tsamardinos I, Brown LE, Aliferis CF (2006) The max–min hill-climbing Bayesian network structure learning algorithm. Mach Learn 65(1):31–78CrossRefGoogle Scholar
  28. Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques. Morgan Kaufmann, San FranciscozbMATHGoogle Scholar

Copyright information

© Springer-Verlag 2010

Authors and Affiliations

  • Juan I. Alonso-Barba
    • 1
  • Luis delaOssa
    • 1
  • Jose M. Puerta
    • 1
  1. 1.Laboratory of Intelligent Systems and Data Mining, Albacete Research Institute of InformaticsUniversity of Castilla-La ManchaAlbaceteSpain

Personalised recommendations