Abstract
Structural learning of Bayesian networks (BNs) is an NP-hard problem which is generally addressed by means of heuristic search algorithms. Despite the fact that earlier proposals for dealing with this task were based on searching the space of Directed Acyclic Graphs (DAGs), there are some alternative approaches. One of these approaches for structural learning consists of searching the space of orderings, as given a certain topological order among the problem variables, it is relatively easy to build (and evaluate) a BN compatible with it. In practice, the latter methods make it possible to obtain good results, but they are still costly in terms of computation. In this article, we prove the correctness of the method used to evaluate each ordering, and we propose some efficient learning algorithms based on it. Our first proposal is based on the Hill-Climbing algorithm, and uses an improved neighbourhood definition. The second algorithm is an extension of the first one, and is based on the well-known Variable Neighbourhood Search metaheuristic. Finally, iterative versions of both algorithms are also proposed. The algorithms have been tested over a set of different domains, and have been compared with other methods such as Hill-Climbing in the space of DAGs or Greedy Equivalent Search, in order to study their behaviour in practice.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We use standard notation, that is, bold font to denote sets and n-dimensional configurations, calligraphic font to denote mathematical structures, upper case for variables or sets of random variables, and lower case to denote states of variables or configurations of states (vectors).
References
Andreassen S, Jensen FV, Andersen SK, Falck B, Kjrulff U, Woldbye M, Srensen AR, Rosenfalck A, Jensen F (1989) MUNIN—an expert EMG assistant, computer-aided electromyography and expert systems, chap 21
Beinlich IA, Suermondt HJ, Chavez RM, Cooper GF (1989) The alarm monitoring system: a case study with two probabilistic inference techniques for belief networks. In: Second European Conference on Artificial Intelligence in Medicine, vol 38. Springer, Berlin, pp 247–256
Binder J, Koller D, Russell SJ, Kanazawa K (1997) Adaptive probabilistic networks with hidden variables. Mach Learn 29(2–3):213–244
Blanco R, Inza I, Larrañaga P (2003) Learning Bayesian networks in the space of structures by estimation of distribution algorithms. Int J Intell Syst 18(2):205–220
Buntine W (1991) Theory refinement on Bayesian networks. In: Proceedings of the seventh conference (1991) on uncertainty in artificial intelligence (UAI’91). Morgan Kaufmann, Los Angeles, pp 52–60
de Campos L, Puerta J (2001) Stochastic local algorithms for learning belief networks: searching in the space of the orderings. In: Symbolic and quantitative approaches to reasoning with uncertainty. Lecture notes in artificial intelligence, vol 2143. Springer, Berlin, pp 228–239
Chickering D (2002) Optimal structure identification with greedy search. J Mach Learn Res 3:507–554
Chickering D, Geiger D, Heckerman D (1995) Learning Bayesian networks: search methods and experimental results. In: Proceedings of the fifth conference on artificial intelligence and statistics, pp 112–128
Chickering DM (1996) Learning Bayesian networks is NP-Complete. In: Fisher D, Lenz HJ (eds) Learning from data: artificial intelligence and statistics V. Springer, Berlin, pp 121–130
Cooper G, Herskovits E (1992) A Bayesian method for the induction of probabilistic networks from data. Mach Learn 9:309–347
Cowell RG, Dawid AP, Lauritzen S, Spiegelhalter D (2003) Probabilistic networks and expert systems (information science and statistics). Springer, Berlin
Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30
Friedman M (1940) A comparison of alternative tests of significance for the problem of m rankings. Ann Math Stat 11(1):86–92
Friedman N, Koller D (2003) Being Bayesian about network structure: a Bayesian approach to structure discovery in Bayesian networks. Mach Learn 50:95–126
Gámez JA, Mateo JL, Puerta JM (2010) Learning Bayesian networks by hill climbing: efficient methods based on progressive restriction of the neighborhood. Data Min Knowl Discov (to appear). doi:10.1007/s10618-010-0178-6
García S, Herrera F (2008) An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons. J Mach Learn Res 9:2677–2694
Heckerman D, Geiger D, Chickering D (1995) Learning Bayesian networks: the combination of knowledge and statistical data. Mach Learn 20(3):197–243
Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6:65–70
Jensen A, Jensen F (1996) Midas—an influece diagram for management of mildew in winter wheat. In: Proceedings of the 12th annual conference on uncertainty in artificial intelligence (UAI-96), pp 349–356
Jensen CS (1997) Blocking Gibbs sampling for inference in large and complex Bayesian networks with applications in genetics. PhD thesis, Aalborg University, Denmark
Jensen F, Nielsen T (2007) Bayesian networks and decision graphs. Springer, Berlin
Kristensen K, Rasmussen IA (2002) The use of a Bayesian network in the design of a decision support system for growing malting barley without use of pesticides. Comput Electron Agric 33:197–217
Larrañaga P, Poza M, Yurramendi Y, Murga R, Kuijpers C (1996) Structure learning of Bayesian networks by genetic algorithms: a performance analysis of control parameters. IEEE Trans Pattern Anal Mach Intell 18(9):912–926
Mladenović N, Hansen P (1997) Variable neighborhood search. Comput Oper Res 24:1097–1100. http://citeseer.ist.psu.edu/mladenovic97variable.html
Pearl J (1988) Probabilistic reasoning in intelligent systems. Morgan Kaufmann, San Mateo
Teyssier M, Koller D (2005) Ordering-based search: a simple and effective algorithm for learning Bayesian networks. In: UAI ’05. Proceedings of the 21st conference in uncertainty in artificial intelligence. AUAI Press, Edinburgh, pp 548–549
Tsamardinos I, Brown LE, Aliferis CF (2006) The max–min hill-climbing Bayesian network structure learning algorithm. Mach Learn 65(1):31–78
Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques. Morgan Kaufmann, San Francisco
Author information
Authors and Affiliations
Corresponding author
Additional information
Research Projects PCI08-0048-8577 and TIN2007-67418-C03-01.
Rights and permissions
About this article
Cite this article
Alonso-Barba, J.I., delaOssa, L. & Puerta, J.M. Structural learning of Bayesian networks using local algorithms based on the space of orderings. Soft Comput 15, 1881–1895 (2011). https://doi.org/10.1007/s00500-010-0623-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-010-0623-x