Advertisement

Swarm Intelligence

, Volume 7, Issue 2–3, pp 229–254 | Cite as

Learning Bayesian network classifiers using ant colony optimization

  • Khalid M. Salama
  • Alex A. Freitas
Article

Abstract

Bayesian networks are knowledge representation tools that model the (in)dependency relationships among variables for probabilistic reasoning. Classification with Bayesian networks aims to compute the class with the highest probability given a case. This special kind is referred to as Bayesian network classifiers. Since learning the Bayesian network structure from a dataset can be viewed as an optimization problem, heuristic search algorithms may be applied to build high-quality networks in medium- or large-scale problems, as exhaustive search is often feasible only for small problems. In this paper, we present our new algorithm, ABC-Miner, and propose several extensions to it. ABC-Miner uses ant colony optimization for learning the structure of Bayesian network classifiers. We report extended computational results comparing the performance of our algorithm with eight other classification algorithms, namely six variations of well-known Bayesian network classifiers, cAnt-Miner for discovering classification rules and a support vector machine algorithm.

Keywords

Ant colony optimization (ACO) Data mining Classification Bayesian network classifiers 

References

  1. Bonabeau, E., Dorigo, M., & Theraulaz, G. (1999). Swarm intelligence: from natural to artificial systems. New York: Oxford University Press. zbMATHGoogle Scholar
  2. Buntine, W. (1991). Theory refinement on Bayesian networks. In 17th conference on uncertainty in artificial intelligence (pp. 52–60). San Francisco: Morgan Kaufmann. Google Scholar
  3. Campos, L. M., Gámez, J. A., & Puerta, J. M. (2002). Ant colony optimization for learning Bayesian networks. Journal of Approximate Reasoning, 31, 291–311. zbMATHCrossRefGoogle Scholar
  4. Cheng, J., & Greiner, R. (1999). Comparing Bayesian network classifiers. In 15th annual conference on uncertainty in artificial intelligence (pp. 101–108). San Francisco: Morgan Kaufmann. Google Scholar
  5. Cheng, J., & Greiner, R. (2001). Learning Bayesian belief network classifiers: algorithms and system. In 14th biennial conference of the Canadian society on computational studies of intelligence: advances in artificial intelligence (pp. 141–151). London: Springer. Google Scholar
  6. Chickering, D., Geiger, M., & Heckerman, D. (1994). Learning Bayesian networks is NP-hard (Technical Report). Advanced Technologies Division, Microsoft Corporation, Redmond, WA. Google Scholar
  7. Colorni, A., Dorigo, M., & Maniezzo, V. (1992). Distributed optimization by ant colonies. In 1st European conference on artificial life (pp. 134–142). Cambridge: MIT Press. Google Scholar
  8. Cooper, G. F., & Herskovits, E. (1992). A Bayesian method for the induction of probabilistic networks from data. Machine Learning, 9, 309–348. zbMATHGoogle Scholar
  9. Cristianini, N., & Shawe-Taylor, J. (2000). An introduction to support vector machines and other kernel-based learning methods. Cambridge: Cambridge University Press. CrossRefGoogle Scholar
  10. Daly, R., & Shen, Q. (2009). Learning Bayesian network equivalence classes with ant colony optimization. Journal of Artificial Intelligence Research, 35, 391–447. MathSciNetzbMATHGoogle Scholar
  11. Daly, R., Shen, Q., & Aitken, S. (2006). Using ant colony optimization in learning Bayesian network equivalence classes. In UK workshop on computational intelligence (UKCI) (pp. 111–118). Palo Alto: AAAI Press. Google Scholar
  12. Demšar, J. (2006). Statistical comparisons of classifiers over multiple datasets. Journal of Machine Learning Research, 7, 1–30. zbMATHGoogle Scholar
  13. Dorigo, M., & Di Caro, G. (1999). The ant colony optimization meta-heuristic. In New ideas in optimization (Vol. 2, pp. 11–32). New York: McGraw-Hill. Google Scholar
  14. Dorigo, M., & Stützle, T. (2004). Ant colony optimization. Cambridge: MIT Press. zbMATHCrossRefGoogle Scholar
  15. Dorigo, M., Maniezzo, V., & Colorni, A. (1996). Ant system: optimization by a colony of cooperating agents. IEEE Transactions on Systems, Man and Cybernetics. Part B. Cybernetics, 26, 29–41. CrossRefGoogle Scholar
  16. Dorigo, M., Di Caro, G., & Gambardella, L. M. (1999). Ant algorithms for discrete optimization. Artificial Life, 5(2), 137–172. CrossRefGoogle Scholar
  17. Freitas, A. A., Wieser, D. C., & Apweiler, R. (2010). On the importance of comprehensible classification models for protein function prediction. ACM/IEEE Transactions on Computational Biology and Bioinformatics, 7, 172–182. CrossRefGoogle Scholar
  18. Friedman, N., & Goldszmidt, M. (1998). Learning Bayesian networks with local structure. In Learning in graphical models (pp. 421–460). Norwell: Kluwer. CrossRefGoogle Scholar
  19. Friedman, N., Geiger, D., & Goldszmidt, M. (1997). Bayesian network classifiers. Machine Learning, 29, 131–161. zbMATHCrossRefGoogle Scholar
  20. García, S., & Herrera, F. (2008). An extension on statistical comparisons of classifiers over multiple datasets for all pairwise comparisons. Journal of Machine Learning Research, 9, 2677–2694. zbMATHGoogle Scholar
  21. Heckerman, D., Geiger, D., & Chickering, D. M. (1995). Learning Bayesian networks: the combination of knowledge and statistical data. Machine Learning, 20, 197–244. zbMATHGoogle Scholar
  22. Huysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., & Baesens, B. (2011). An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models. Decision Support Systems, 51, 141–154. CrossRefGoogle Scholar
  23. Jaiwei, H., & Kamber, M. (2006). Data mining: concepts and techniques. San Francisco: Morgan Kaufmann. Google Scholar
  24. Ji, J., Hu, R., Zhang, H., & Liu, C. (2011). A hybrid method for learning Bayesian networks based on ant colony optimization. Applied Soft Computing, 11, 3373–3384. CrossRefGoogle Scholar
  25. Jiang, L., Wang, D., Cai, Z., & Yan, X. (2007). Survey of improving naive Bayes for classification. In LNCS: Vol. 4632. International conference on advanced data mining and applications (pp. 134–145). Heidelberg: Springer. CrossRefGoogle Scholar
  26. Keogh, E., & Pazzani, M. (1999). Learning augmented Bayesian classifiers: a comparison of distribution-based and classification-based approaches. In International workshop on artificial intelligence and statistics (pp. 225–230). San Francisco: Morgan Kaufmann. Google Scholar
  27. Kohavi, R., & Sahami, M. (1996). Error-based and entropy-based discretization of continuous features. In International ACM SIGKDD conference on knowledge discovery and data mining (pp. 114–119). Palo Alto: AAAI Press. Google Scholar
  28. Langley, P., Iba, W., & Thompson, K. (1992). An analysis of Bayesian classifiers. In 10th national conference on artificial intelligence (AAAI-92) (pp. 223–228). Palo Alto: AAAI Press. Google Scholar
  29. Marteens, D., Vanthienen, J., Verbeke, W., & Baesens, B. (2011). Performance of classification models from a user perspective. Decision Support Systems, 51, 782–793. CrossRefGoogle Scholar
  30. Martens, D., Backer, M. D., Haesen, R., Vanthienen, J., Snoeck, M., & Baesens, B. (2007). Classification with ant colony optimization. IEEE Transactions on Evolutionary Computation, 11, 651–665. CrossRefGoogle Scholar
  31. Martens, D., Baesens, B., & Fawcett, T. (2011). Editorial survey: swarm intelligence for data mining. Machine Learning, 82, 1–42. CrossRefGoogle Scholar
  32. Otero, F., Freitas, A. A., & Johnson, C.G. (2008). cAnt-Miner: an ant colony classification algorithm to cope with continuous attributes. In LNCS: Vol. 5217. Sixth international conference on ant colony optimization and swarm intelligence (pp. 48–59). Heidelberg: Springer. CrossRefGoogle Scholar
  33. Parpinelli, R. S., Lopes, H. S., & Freitas, A. A. (2002). Data mining with an ant colony optimization algorithm. IEEE Transactions on Evolutionary Computation, 6(4), 321–332. CrossRefGoogle Scholar
  34. Pazzani, M. J., Mani, S., & Shankle, W. R. (2001). Acceptance of rules generated by machine learning among medical experts. Methods of Information in Medicine, 40, 380–385. Google Scholar
  35. Pinto, P. C., Nägele, A., Dejori, M., Runkler, T. A., & Costa, J. M. (2008). Learning of Bayesian networks by a local discovery ant colony algorithm. In IEEE world congress on computational intelligence (pp. 2741–2748). Piscataway: IEEE Press. Google Scholar
  36. Pinto, P. C., Nägele, A., Dejori, M., Runkler, T. A., & Costa, J. M. (2009). Using a local discovery ant algorithm for Bayesian network structure learning. IEEE Transactions on Evolutionary Computation, 13, 767–779. CrossRefGoogle Scholar
  37. Salama, K. M., & Abdelbar, A. M. (2010). Extensions to the Ant-Miner classification rule discovery algorithm. In LNCS: Vol. 6234. 7th international conference on swarm intelligence (ANTS 2010) (pp. 43–50). Heidelberg: Springer. Google Scholar
  38. Salama, K. M., & Abdelbar, A. M. (2011). Exploring different rule quality evaluation functions in ACO-based classification algorithms. In IEEE symposium on swarm intelligence (SIS) (pp. 1–8). Piscataway: IEEE Press. Google Scholar
  39. Salama, K. M., & Freitas, A. A. (2012). ABC-Miner: an ant-based Bayesian classification algorithm. In LNCS: Vol. 7461. 8th international conference on swarm intelligence (ANTS 2012) (pp. 13–24). Heidelberg: Springer. Google Scholar
  40. Salama, K. M., Abdelbar, A. M., & Freitas, A. A. (2011). Multiple pheromone types and other extensions to the Ant-Miner classification rule discovery algorithm. Swarm Intelligence, 5, 149–182. CrossRefGoogle Scholar
  41. Salama, K. M., Abdelbar, A. M., Otero, F. E., & Freitas, A. A. (2013). Utilizing multiple pheromones in an ant-based algorithm for continuous-attribute classification rule discovery. Applied Soft Computing, 13, 667–675. CrossRefGoogle Scholar
  42. UCI Repository of machine learning databases. Retrieved Oct. 2011 from. http://www.ics.uci.edu/~mlearn/MLRepository.html.
  43. Witten, H., & Frank, E. (2005). Data mining: practical machine learning tools and techniques (2nd ed.). San Francisco: Morgan Kauffman. Google Scholar
  44. Yang, S. (2002). Comparison of score metrics for Bayesian network learning. IEEE Transactions on Systems, Man and Cybernetics. Part A. Systems and Humans, 32, 419–428. CrossRefGoogle Scholar
  45. Yanghui, Wu., McCall, J., & Corne, D. (2010). Two novel ant colony optimization approaches for Bayesian network structure learning. In IEEE world congress on evolutionary computation (CEC 2010) (pp. 1–7). Piscataway: IEEE Press. Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.School of ComputingUniversity of KentCanterburyUK

Personalised recommendations