Skip to main content
SpringerLink
Log in

Learning Bayesian network classifiers using ant colony optimization

  • Published:
Swarm Intelligence Aims and scope Submit manuscript

Abstract

Bayesian networks are knowledge representation tools that model the (in)dependency relationships among variables for probabilistic reasoning. Classification with Bayesian networks aims to compute the class with the highest probability given a case. This special kind is referred to as Bayesian network classifiers. Since learning the Bayesian network structure from a dataset can be viewed as an optimization problem, heuristic search algorithms may be applied to build high-quality networks in medium- or large-scale problems, as exhaustive search is often feasible only for small problems. In this paper, we present our new algorithm, ABC-Miner, and propose several extensions to it. ABC-Miner uses ant colony optimization for learning the structure of Bayesian network classifiers. We report extended computational results comparing the performance of our algorithm with eight other classification algorithms, namely six variations of well-known Bayesian network classifiers, cAnt-Miner for discovering classification rules and a support vector machine algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Algorithm 2
Algorithm 3
Fig. 2

Similar content being viewed by others

References

  • Bonabeau, E., Dorigo, M., & Theraulaz, G. (1999). Swarm intelligence: from natural to artificial systems. New York: Oxford University Press.

    MATH  Google Scholar 

  • Buntine, W. (1991). Theory refinement on Bayesian networks. In 17th conference on uncertainty in artificial intelligence (pp. 52–60). San Francisco: Morgan Kaufmann.

    Google Scholar 

  • Campos, L. M., Gámez, J. A., & Puerta, J. M. (2002). Ant colony optimization for learning Bayesian networks. Journal of Approximate Reasoning, 31, 291–311.

    Article  MATH  Google Scholar 

  • Cheng, J., & Greiner, R. (1999). Comparing Bayesian network classifiers. In 15th annual conference on uncertainty in artificial intelligence (pp. 101–108). San Francisco: Morgan Kaufmann.

    Google Scholar 

  • Cheng, J., & Greiner, R. (2001). Learning Bayesian belief network classifiers: algorithms and system. In 14th biennial conference of the Canadian society on computational studies of intelligence: advances in artificial intelligence (pp. 141–151). London: Springer.

    Google Scholar 

  • Chickering, D., Geiger, M., & Heckerman, D. (1994). Learning Bayesian networks is NP-hard (Technical Report). Advanced Technologies Division, Microsoft Corporation, Redmond, WA.

  • Colorni, A., Dorigo, M., & Maniezzo, V. (1992). Distributed optimization by ant colonies. In 1st European conference on artificial life (pp. 134–142). Cambridge: MIT Press.

    Google Scholar 

  • Cooper, G. F., & Herskovits, E. (1992). A Bayesian method for the induction of probabilistic networks from data. Machine Learning, 9, 309–348.

    MATH  Google Scholar 

  • Cristianini, N., & Shawe-Taylor, J. (2000). An introduction to support vector machines and other kernel-based learning methods. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Daly, R., & Shen, Q. (2009). Learning Bayesian network equivalence classes with ant colony optimization. Journal of Artificial Intelligence Research, 35, 391–447.

    MathSciNet  MATH  Google Scholar 

  • Daly, R., Shen, Q., & Aitken, S. (2006). Using ant colony optimization in learning Bayesian network equivalence classes. In UK workshop on computational intelligence (UKCI) (pp. 111–118). Palo Alto: AAAI Press.

    Google Scholar 

  • Demšar, J. (2006). Statistical comparisons of classifiers over multiple datasets. Journal of Machine Learning Research, 7, 1–30.

    MATH  Google Scholar 

  • Dorigo, M., & Di Caro, G. (1999). The ant colony optimization meta-heuristic. In New ideas in optimization (Vol. 2, pp. 11–32). New York: McGraw-Hill.

    Google Scholar 

  • Dorigo, M., & Stützle, T. (2004). Ant colony optimization. Cambridge: MIT Press.

    Book  MATH  Google Scholar 

  • Dorigo, M., Maniezzo, V., & Colorni, A. (1996). Ant system: optimization by a colony of cooperating agents. IEEE Transactions on Systems, Man and Cybernetics. Part B. Cybernetics, 26, 29–41.

    Article  Google Scholar 

  • Dorigo, M., Di Caro, G., & Gambardella, L. M. (1999). Ant algorithms for discrete optimization. Artificial Life, 5(2), 137–172.

    Article  Google Scholar 

  • Freitas, A. A., Wieser, D. C., & Apweiler, R. (2010). On the importance of comprehensible classification models for protein function prediction. ACM/IEEE Transactions on Computational Biology and Bioinformatics, 7, 172–182.

    Article  Google Scholar 

  • Friedman, N., & Goldszmidt, M. (1998). Learning Bayesian networks with local structure. In Learning in graphical models (pp. 421–460). Norwell: Kluwer.

    Chapter  Google Scholar 

  • Friedman, N., Geiger, D., & Goldszmidt, M. (1997). Bayesian network classifiers. Machine Learning, 29, 131–161.

    Article  MATH  Google Scholar 

  • García, S., & Herrera, F. (2008). An extension on statistical comparisons of classifiers over multiple datasets for all pairwise comparisons. Journal of Machine Learning Research, 9, 2677–2694.

    MATH  Google Scholar 

  • Heckerman, D., Geiger, D., & Chickering, D. M. (1995). Learning Bayesian networks: the combination of knowledge and statistical data. Machine Learning, 20, 197–244.

    MATH  Google Scholar 

  • Huysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., & Baesens, B. (2011). An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models. Decision Support Systems, 51, 141–154.

    Article  Google Scholar 

  • Jaiwei, H., & Kamber, M. (2006). Data mining: concepts and techniques. San Francisco: Morgan Kaufmann.

    Google Scholar 

  • Ji, J., Hu, R., Zhang, H., & Liu, C. (2011). A hybrid method for learning Bayesian networks based on ant colony optimization. Applied Soft Computing, 11, 3373–3384.

    Article  Google Scholar 

  • Jiang, L., Wang, D., Cai, Z., & Yan, X. (2007). Survey of improving naive Bayes for classification. In LNCS: Vol. 4632. International conference on advanced data mining and applications (pp. 134–145). Heidelberg: Springer.

    Chapter  Google Scholar 

  • Keogh, E., & Pazzani, M. (1999). Learning augmented Bayesian classifiers: a comparison of distribution-based and classification-based approaches. In International workshop on artificial intelligence and statistics (pp. 225–230). San Francisco: Morgan Kaufmann.

    Google Scholar 

  • Kohavi, R., & Sahami, M. (1996). Error-based and entropy-based discretization of continuous features. In International ACM SIGKDD conference on knowledge discovery and data mining (pp. 114–119). Palo Alto: AAAI Press.

    Google Scholar 

  • Langley, P., Iba, W., & Thompson, K. (1992). An analysis of Bayesian classifiers. In 10th national conference on artificial intelligence (AAAI-92) (pp. 223–228). Palo Alto: AAAI Press.

    Google Scholar 

  • Marteens, D., Vanthienen, J., Verbeke, W., & Baesens, B. (2011). Performance of classification models from a user perspective. Decision Support Systems, 51, 782–793.

    Article  Google Scholar 

  • Martens, D., Backer, M. D., Haesen, R., Vanthienen, J., Snoeck, M., & Baesens, B. (2007). Classification with ant colony optimization. IEEE Transactions on Evolutionary Computation, 11, 651–665.

    Article  Google Scholar 

  • Martens, D., Baesens, B., & Fawcett, T. (2011). Editorial survey: swarm intelligence for data mining. Machine Learning, 82, 1–42.

    Article  Google Scholar 

  • Otero, F., Freitas, A. A., & Johnson, C.G. (2008). cAnt-Miner: an ant colony classification algorithm to cope with continuous attributes. In LNCS: Vol. 5217. Sixth international conference on ant colony optimization and swarm intelligence (pp. 48–59). Heidelberg: Springer.

    Chapter  Google Scholar 

  • Parpinelli, R. S., Lopes, H. S., & Freitas, A. A. (2002). Data mining with an ant colony optimization algorithm. IEEE Transactions on Evolutionary Computation, 6(4), 321–332.

    Article  Google Scholar 

  • Pazzani, M. J., Mani, S., & Shankle, W. R. (2001). Acceptance of rules generated by machine learning among medical experts. Methods of Information in Medicine, 40, 380–385.

    Google Scholar 

  • Pinto, P. C., Nägele, A., Dejori, M., Runkler, T. A., & Costa, J. M. (2008). Learning of Bayesian networks by a local discovery ant colony algorithm. In IEEE world congress on computational intelligence (pp. 2741–2748). Piscataway: IEEE Press.

    Google Scholar 

  • Pinto, P. C., Nägele, A., Dejori, M., Runkler, T. A., & Costa, J. M. (2009). Using a local discovery ant algorithm for Bayesian network structure learning. IEEE Transactions on Evolutionary Computation, 13, 767–779.

    Article  Google Scholar 

  • Salama, K. M., & Abdelbar, A. M. (2010). Extensions to the Ant-Miner classification rule discovery algorithm. In LNCS: Vol. 6234. 7th international conference on swarm intelligence (ANTS 2010) (pp. 43–50). Heidelberg: Springer.

    Google Scholar 

  • Salama, K. M., & Abdelbar, A. M. (2011). Exploring different rule quality evaluation functions in ACO-based classification algorithms. In IEEE symposium on swarm intelligence (SIS) (pp. 1–8). Piscataway: IEEE Press.

    Google Scholar 

  • Salama, K. M., & Freitas, A. A. (2012). ABC-Miner: an ant-based Bayesian classification algorithm. In LNCS: Vol. 7461. 8th international conference on swarm intelligence (ANTS 2012) (pp. 13–24). Heidelberg: Springer.

    Google Scholar 

  • Salama, K. M., Abdelbar, A. M., & Freitas, A. A. (2011). Multiple pheromone types and other extensions to the Ant-Miner classification rule discovery algorithm. Swarm Intelligence, 5, 149–182.

    Article  Google Scholar 

  • Salama, K. M., Abdelbar, A. M., Otero, F. E., & Freitas, A. A. (2013). Utilizing multiple pheromones in an ant-based algorithm for continuous-attribute classification rule discovery. Applied Soft Computing, 13, 667–675.

    Article  Google Scholar 

  • UCI Repository of machine learning databases. Retrieved Oct. 2011 from. http://www.ics.uci.edu/~mlearn/MLRepository.html.

  • Witten, H., & Frank, E. (2005). Data mining: practical machine learning tools and techniques (2nd ed.). San Francisco: Morgan Kauffman.

    Google Scholar 

  • Yang, S. (2002). Comparison of score metrics for Bayesian network learning. IEEE Transactions on Systems, Man and Cybernetics. Part A. Systems and Humans, 32, 419–428.

    Article  Google Scholar 

  • Yanghui, Wu., McCall, J., & Corne, D. (2010). Two novel ant colony optimization approaches for Bayesian network structure learning. In IEEE world congress on evolutionary computation (CEC 2010) (pp. 1–7). Piscataway: IEEE Press.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing, University of Kent, Canterbury, CT2 7NF, UK

    Khalid M. Salama & Alex A. Freitas

Authors
  1. Khalid M. Salama
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alex A. Freitas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Khalid M. Salama.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Salama, K.M., Freitas, A.A. Learning Bayesian network classifiers using ant colony optimization. Swarm Intell 7, 229–254 (2013). https://doi.org/10.1007/s11721-013-0087-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11721-013-0087-6

Keywords

Search

Navigation