# Structural learning of Bayesian networks using local algorithms based on the space of orderings

- 202 Downloads
- 5 Citations

## Abstract

Structural learning of Bayesian networks (BNs) is an NP-hard problem which is generally addressed by means of heuristic search algorithms. Despite the fact that earlier proposals for dealing with this task were based on searching the space of Directed Acyclic Graphs (DAGs), there are some alternative approaches. One of these approaches for structural learning consists of searching the space of orderings, as given a certain topological order among the problem variables, it is relatively easy to build (and evaluate) a BN compatible with it. In practice, the latter methods make it possible to obtain good results, but they are still costly in terms of computation. In this article, we prove the correctness of the method used to evaluate each ordering, and we propose some efficient learning algorithms based on it. Our first proposal is based on the Hill-Climbing algorithm, and uses an improved neighbourhood definition. The second algorithm is an extension of the first one, and is based on the well-known Variable Neighbourhood Search metaheuristic. Finally, iterative versions of both algorithms are also proposed. The algorithms have been tested over a set of different domains, and have been compared with other methods such as Hill-Climbing in the space of DAGs or *Greedy Equivalent Search*, in order to study their behaviour in practice.

## Keywords

Bayesian networks Structural learning Orderings Variable Neighbourhood Search## References

- Andreassen S, Jensen FV, Andersen SK, Falck B, Kjrulff U, Woldbye M, Srensen AR, Rosenfalck A, Jensen F (1989) MUNIN—an expert EMG assistant, computer-aided electromyography and expert systems, chap 21Google Scholar
- Beinlich IA, Suermondt HJ, Chavez RM, Cooper GF (1989) The alarm monitoring system: a case study with two probabilistic inference techniques for belief networks. In: Second European Conference on Artificial Intelligence in Medicine, vol 38. Springer, Berlin, pp 247–256Google Scholar
- Binder J, Koller D, Russell SJ, Kanazawa K (1997) Adaptive probabilistic networks with hidden variables. Mach Learn 29(2–3):213–244MATHCrossRefGoogle Scholar
- Blanco R, Inza I, Larrañaga P (2003) Learning Bayesian networks in the space of structures by estimation of distribution algorithms. Int J Intell Syst 18(2):205–220MATHCrossRefGoogle Scholar
- Buntine W (1991) Theory refinement on Bayesian networks. In: Proceedings of the seventh conference (1991) on uncertainty in artificial intelligence (UAI’91). Morgan Kaufmann, Los Angeles, pp 52–60Google Scholar
- de Campos L, Puerta J (2001) Stochastic local algorithms for learning belief networks: searching in the space of the orderings. In: Symbolic and quantitative approaches to reasoning with uncertainty. Lecture notes in artificial intelligence, vol 2143. Springer, Berlin, pp 228–239Google Scholar
- Chickering D (2002) Optimal structure identification with greedy search. J Mach Learn Res 3:507–554MathSciNetGoogle Scholar
- Chickering D, Geiger D, Heckerman D (1995) Learning Bayesian networks: search methods and experimental results. In: Proceedings of the fifth conference on artificial intelligence and statistics, pp 112–128Google Scholar
- Chickering DM (1996) Learning Bayesian networks is NP-Complete. In: Fisher D, Lenz HJ (eds) Learning from data: artificial intelligence and statistics V. Springer, Berlin, pp 121–130Google Scholar
- Cooper G, Herskovits E (1992) A Bayesian method for the induction of probabilistic networks from data. Mach Learn 9:309–347MATHGoogle Scholar
- Cowell RG, Dawid AP, Lauritzen S, Spiegelhalter D (2003) Probabilistic networks and expert systems (information science and statistics). Springer, BerlinGoogle Scholar
- Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MathSciNetGoogle Scholar
- Friedman M (1940) A comparison of alternative tests of significance for the problem of
*m*rankings. Ann Math Stat 11(1):86–92MATHCrossRefGoogle Scholar - Friedman N, Koller D (2003) Being Bayesian about network structure: a Bayesian approach to structure discovery in Bayesian networks. Mach Learn 50:95–126MATHCrossRefGoogle Scholar
- Gámez JA, Mateo JL, Puerta JM (2010) Learning Bayesian networks by hill climbing: efficient methods based on progressive restriction of the neighborhood. Data Min Knowl Discov (to appear). doi: 10.1007/s10618-010-0178-6
- García S, Herrera F (2008) An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons. J Mach Learn Res 9:2677–2694Google Scholar
- Heckerman D, Geiger D, Chickering D (1995) Learning Bayesian networks: the combination of knowledge and statistical data. Mach Learn 20(3):197–243MATHGoogle Scholar
- Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6:65–70MathSciNetMATHGoogle Scholar
- Jensen A, Jensen F (1996) Midas—an influece diagram for management of mildew in winter wheat. In: Proceedings of the 12th annual conference on uncertainty in artificial intelligence (UAI-96), pp 349–356Google Scholar
- Jensen CS (1997) Blocking Gibbs sampling for inference in large and complex Bayesian networks with applications in genetics. PhD thesis, Aalborg University, DenmarkGoogle Scholar
- Jensen F, Nielsen T (2007) Bayesian networks and decision graphs. Springer, BerlinGoogle Scholar
- Kristensen K, Rasmussen IA (2002) The use of a Bayesian network in the design of a decision support system for growing malting barley without use of pesticides. Comput Electron Agric 33:197–217CrossRefGoogle Scholar
- Larrañaga P, Poza M, Yurramendi Y, Murga R, Kuijpers C (1996) Structure learning of Bayesian networks by genetic algorithms: a performance analysis of control parameters. IEEE Trans Pattern Anal Mach Intell 18(9):912–926CrossRefGoogle Scholar
- Mladenović N, Hansen P (1997) Variable neighborhood search. Comput Oper Res 24:1097–1100. http://citeseer.ist.psu.edu/mladenovic97variable.html Google Scholar
- Pearl J (1988) Probabilistic reasoning in intelligent systems. Morgan Kaufmann, San MateoGoogle Scholar
- Teyssier M, Koller D (2005) Ordering-based search: a simple and effective algorithm for learning Bayesian networks. In: UAI ’05. Proceedings of the 21st conference in uncertainty in artificial intelligence. AUAI Press, Edinburgh, pp 548–549Google Scholar
- Tsamardinos I, Brown LE, Aliferis CF (2006) The max–min hill-climbing Bayesian network structure learning algorithm. Mach Learn 65(1):31–78CrossRefGoogle Scholar
- Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques. Morgan Kaufmann, San FranciscoMATHGoogle Scholar