Empirical Behavior of Bayesian Network Structure Learning Algorithms

  • Brandon Malone
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9505)


Bayesian network structure learning (BNSL) is the problem of finding a BN structure which best explains a dataset. Score-based learning assigns a score to each network structure. The goal is to find the structure which optimizes the score. We review two recent studies of empirical behavior of BNSL algorithms.

The score typically reflects fit to a training dataset; however, models which fit training data well may generalize poorly. Thus, it is not clear that finding an optimal network is worthwhile. We review a comparison of exact and approximate search techniques. Sometimes, approximate algorithms suffice; for complex datasets, the optimal algorithms produce better networks.

BNSL is known to be NP-hard, so exact solvers prune the search space using heuristics. We next review problem-dependent characteristics which affect their efficacy. Empirical results show that machine learning techniques based on these characteristics can often be used to accurately predict the algorithms’ running times.


Bayesian networks Structure learning Algorithm selection Empirical hardness 


  1. Bartlett, M., Cussens, J.: Integer linear programming for the Bayesian network structure learning problem. In: Artificial Intelligence (2015)Google Scholar
  2. Chickering, D.M.: Learning Bayesian networks is NP-complete. In: Fisher, D., Lenz, H.-J. (eds.) Learning from Data: Artificial Intelligence and Statistics V. Lecture Notes in Statistics, vol. 112, pp. 121–130. Springer, New York (1996)CrossRefGoogle Scholar
  3. Chickering, D.M.: Learning equivalence classes of Bayesian-network structures. J. Mach. Learn. Res. 2, 445–498 (2002)MathSciNetzbMATHGoogle Scholar
  4. Chow, C., Liu, C.: Approximating discrete probability distributions with dependence trees. IEEE Trans. Inf. Theory 14(3), 462–467 (1968)CrossRefGoogle Scholar
  5. Cooper, G.F., Herskovits, E.: A Bayesian method for the induction of probabilistic networks from data. Mach. Learn. 9, 309–347 (1992)zbMATHGoogle Scholar
  6. Cussens, J., Bartlett, M., Jones, E.M., Sheehan, N.A.: Maximum likelihood pedigree reconstruction using integer linear programming. Genet. Epidemiol. 37(1), 69–83 (2013)CrossRefGoogle Scholar
  7. de Campos, C.P., Ji, Q.: Efficient learning of Bayesian networks using constraints. J. Mach. Learn. Res. 12, 663–689 (2011)MathSciNetzbMATHGoogle Scholar
  8. de Campos, L.M., Huete, J.F.: A new approach for learning belief networks using independence criteria. Int. J. Approximate Reasoning 24(1), 11–37 (2000)MathSciNetCrossRefGoogle Scholar
  9. Fan, X., Yuan, C., Malone, B.: Tightening bounds for Bayesian network structure learning. In: Proceedings of the 28th AAAI Conference on Artificial Intelligence (2014)Google Scholar
  10. Friedman, N., Nachman, I., Peer, D.: Learning Bayesian network structure from massive datasets: the “sparse candidate” algorithm. In: Proceedings 13th Conference on Uncertainty in Artificial Intelligence (1999)Google Scholar
  11. Glover, F.: Tabu search: a tutorial. Interfaces 20(4), 74–94 (1990)CrossRefGoogle Scholar
  12. Heckerman, D., Geiger, D., Chickering, D.M.: Learning Bayesian networks: the combination of knowledge and statistical data. Mach. Learn. 20, 197–243 (1995)zbMATHGoogle Scholar
  13. Koivisto, M., Sood, K.: Exact Bayesian structure discovery in Bayesian networks. J. Mach. Learn. Res. 5, 549–573 (2004)MathSciNetzbMATHGoogle Scholar
  14. Leyton-Brown, K., Nudelman, E., Shoham, Y.: Learning the empirical hardness of optimization problems: the case of combinatorial auctions. In: Hentenryck, P. (ed.) CP 2002. LNCS, vol. 2470, pp. 556–572. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  15. Malone, B., Järvisalo, M., Myllymäki, P.: Impact of learning strategies on the qualpacking Bayesian networks: an empirical evaluation. In: Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence (2015)Google Scholar
  16. Malone, B., Kangas, K., Järvisalo, M., Koivisto, M., Myllymäki, P.: Predicting the hardness of learning Bayesian networks. In: Proceedings of the 28th AAAI Conference on Artificial Intelligence (2014)Google Scholar
  17. Malone, B., Yuan, C.: Evaluating anytime algorithms for learning optimal Bayesian networks. In: Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence (2013)Google Scholar
  18. Mitchell, T.: Machine Learning. McGraw-Hill, New York (1997)zbMATHGoogle Scholar
  19. Moore, A., Wong, W.-K.: Optimal reinsertion: a new search operator for accelerated and more accurate Bayesian network structure learning. In: Proceedings of the 20th International Conference on Machine Learning, pp. 552–559 (2003)Google Scholar
  20. Ott, S., Imoto, S., Miyano, S.: Finding optimal models for small gene networks. In: Proceedings of the Pacific Symposium on Biocomputing (2004)Google Scholar
  21. Parviainen, P., Koivisto, M.: Exact structure discovery in Bayesian networks with less space. In: Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, Quebec, Canada. AUAI Press (2009)Google Scholar
  22. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann Publishers Inc., San Mateo (1988)zbMATHGoogle Scholar
  23. Quinlan, J.R.: Simplifying decision trees. Int. J. Man Mach. Stud. 27, 221–234 (1987)CrossRefGoogle Scholar
  24. Rice, J.R.: The algorithm selection problem. Adv. Comput. 15, 65–118 (1976)CrossRefGoogle Scholar
  25. Russell, S.J., Norvig, P.: Artificial Intelligence: A Modern Approach. Pearson Education, Upper Saddle River (2003)zbMATHGoogle Scholar
  26. Silander, T., Myllymäki, P.: A simple approach for finding the globally optimal Bayesian network structure. In: Proceedings of the 22nd Conference on Uncertainty in Artificial Intelligence (2006)Google Scholar
  27. Silander, T., Roos, T., Kontkanen, P., Myllymäki, P.: Factorized normalized maximum likelihood criterion for learning Bayesian network structures. In: Proceedings of the 4th European Workshop on Probabilistic Graphical Models (2008)Google Scholar
  28. Spirtes, P., Glymour, C., Schemes, R.: Causation, Prediction, and Search, 2nd edn. MIT Press, Cambridge (2000)zbMATHGoogle Scholar
  29. Suzuki, J.: Learning Bayesian belief networks based on the MDL principle: an efficient algorithm using the branch and bound technique. IEICE Trans. Inf. Syst. E82–D(2), 356–367 (1999)Google Scholar
  30. Teyssier, M., Koller, D.: Ordering-based search: a simple and effective algorithm for learning Bayesian networks. In: Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence (2005)Google Scholar
  31. Tian, J.: A branch-and-bound algorithm for MDL learning Bayesian networks. In: Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence (2000)Google Scholar
  32. Tsamardinos, I., Brown, L., Aliferis, C.: The max-min hill-climbing Bayesian network structure learning algorithm. Mach. Learn. 65, 31–78 (2006)CrossRefGoogle Scholar
  33. van Beek, P., Hoffmann, H.-F.: Machine learning of Bayesian networks using constraint programming. In: Pesant, G. (ed.) CP 2015. LNCS, vol. 9255, pp. 429–445. Springer, Heidelberg (2015)Google Scholar
  34. Yuan, C., Malone, B.: Learning optimal Bayesian networks: a shortest path perspective. J. Artif. Intell. Res. 48, 23–65 (2013)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Open Access This chapter is distributed under the terms of the Creative Commons Attribution Noncommercial License, which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Authors and Affiliations

  1. 1.Max Planck Institute for the Biology of AgeingCologneGermany

Personalised recommendations