A Depth-First Branch and Bound Algorithm for Learning Optimal Bayesian Networks

  • Brandon Malone
  • Changhe Yuan
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8323)


Early methods for learning a Bayesian network that optimizes a scoring function for a given dataset are mostly approximation algorithms such as greedy hill climbing approaches. These methods are anytime algorithms as they can be stopped anytime to produce the best solution so far. However, they cannot guarantee the quality of their solution, not even mentioning optimality. In recent years, several exact algorithms have been developed for learning optimal Bayesian network structures. Most of these algorithms only find a solution at the end of the search, so they fail to find any solution if stopped early for some reason, e.g., out of time or memory. We present a new depth-first branch and bound algorithm that finds increasingly better solutions and eventually converges to an optimal Bayesian network upon completion. The algorithm is shown to not only improve the runtime to find optimal network structures up to 100 times compared to some existing methods, but also prove the optimality of these solutions about 10 times faster in some cases.


Bayesian Network Goal Node Minimum Description Length Good Path Heuristic Function 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Cooper, G.F., Herskovits, E.: A Bayesian method for the induction of probabilistic networks from data. Machine Learning 9, 309–347 (1992)zbMATHGoogle Scholar
  2. 2.
    Heckerman, D.: A tutorial on learning with Bayesian networks. In: Jordan, M. (ed.) Learning in Graphical Models. NATO ASI Series, vol. 89, pp. 301–354. Springer, Netherlands (1998)CrossRefGoogle Scholar
  3. 3.
    Moore, A., Wong, W.K.: Optimal reinsertion: A new search operator for accelerated and more accurate Bayesian network structure learning. In: Proceedings of the International Conference on Machine Learning, pp. 552–559 (2003)Google Scholar
  4. 4.
    Koivisto, M., Sood, K.: Exact Bayesian structure discovery in Bayesian networks. Journal of Machine Learning Research, 549–573 (2004)Google Scholar
  5. 5.
    Ott, S., Imoto, S., Miyano, S.: Finding optimal models for small gene networks. In: Pacific Symposium on Biocomputing, pp. 557–567 (2004)Google Scholar
  6. 6.
    Silander, T., Myllymaki, P.: A simple approach for finding the globally optimal Bayesian network structure. In: Proceedings of the 22nd Conference on Uncertainty in Artificial Intelligence. AUAI Press, Arlington (2006)Google Scholar
  7. 7.
    Singh, A., Moore, A.: Finding optimal Bayesian networks by dynamic programming. Technical report, Carnegie Mellon University (June 2005)Google Scholar
  8. 8.
    Parviainen, P., Koivisto, M.: Exact structure discovery in Bayesian networks with less space. In: Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (2009)Google Scholar
  9. 9.
    Malone, B., Yuan, C., Hansen, E.: Memory-efficient dynamic programming for learning optimal Bayesian networks. In: Proceedings of the 25th National Conference on AI (2011)Google Scholar
  10. 10.
    de Campos, C.P., Ji, Q.: Efficient learning of Bayesian networks using constraints. Journal of Machine Learning Research 12, 663–689 (2011)Google Scholar
  11. 11.
    Cussens, J.: Bayesian network learning with cutting planes. In: Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, pp. 153–160. AUAI Press (2011)Google Scholar
  12. 12.
    Jaakkola, T., Sontag, D., Globerson, A., Meila, M.: Learning Bayesian network structure using LP relaxations. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (2010)Google Scholar
  13. 13.
    Yuan, C., Malone, B., Wu, X.: Learning optimal Bayesian networks using A* search. In: Proceedings of the 22nd International Joint Conference on Artificial Intelligence (2011)Google Scholar
  14. 14.
    Malone, B., Yuan, C., Hansen, E., Bridges, S.: Improving the scalability of optimal Bayesian network learning with external-memory frontier breadth-first branch and bound search. In: Proceedings of the 27th Annual Conference on Uncertainty in Artificial Intelligence (2011)Google Scholar
  15. 15.
    Yuan, C., Malone, B.: An improved admissible heuristic for finding optimal Bayesian networks. In: Proceedings of the 28th Conference on Uncertainty in AI (2012)Google Scholar
  16. 16.
    Lam, W., Bacchus, F.: Learning Bayesian belief networks: An approach based on the MDL principle. Computational Intelligence 10, 269–293 (1994)CrossRefGoogle Scholar
  17. 17.
    Tian, J.: A branch-and-bound algorithm for MDL learning Bayesian networks. In: Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, pp. 580–588. Morgan Kaufmann Publishers Inc. (2000)Google Scholar
  18. 18.
    Schwarz, G.: Estimating the dimension of a model. The Annals of Statistics 6, 461–464 (1978)CrossRefzbMATHMathSciNetGoogle Scholar
  19. 19.
    Akaike, H.: Information theory and an extension of the maximum likelihood principle. In: Proceedings of the Second International Symposium on Information Theory, pp. 267–281 (1973)Google Scholar
  20. 20.
    Buntine, W.: Theory refinement on Bayesian networks. In: Proceedings of the Seventh Conference on Uncertainty in Artificial Intelligence, pp. 52–60. Morgan Kaufmann Publishers Inc., San Francisco (1991)Google Scholar
  21. 21.
    Heckerman, D., Geiger, D., Chickering, D.M.: Learning Bayesian networks: The combination of knowledge and statistical data. Machine Learning 20, 197–243 (1995)zbMATHGoogle Scholar
  22. 22.
    Silander, T., Roos, T., Kontkanen, P., Myllymaki, P.: Factorized normalized maximum likelihood criterion for learning Bayesian network structures. In: Proceedings of the 4th European Workshop on Probabilistic Graphical Models (PGM 2008), pp. 257–272 (2008)Google Scholar
  23. 23.
    Teyssier, M., Koller, D.: Ordering-based search: A simple and effective algorithm for learning Bayesian networks. In: Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, pp. 584–590. AUAI Press, Arlington (2005)Google Scholar
  24. 24.
    Frank, A., Asuncion, A.: UCI machine learning repository (2010)Google Scholar
  25. 25.
    Tsamardinos, I., Brown, L., Aliferis, C.: The max-min hill-climbing Bayesian network structure learning algorithm. Machine Learning 65, 31–78 (2006), 10.1007/s10994-006-6889-7Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Brandon Malone
    • 1
  • Changhe Yuan
    • 2
  1. 1.Helsinki Institute for Information Technology, Department of Computer ScienceUniversity of HelsinkiFinland
  2. 2.Queens College/City University of New YorkUSA

Personalised recommendations