Swarm Intelligence

, Volume 11, Issue 3–4, pp 211–242 | Cite as

Learning cluster-based classification systems with ant colony optimization algorithms

Article
  • 298 Downloads

Abstract

Classification is a data mining task the goal of which is to learn a model, from a training dataset, that can predict the class of a new data instance, while clustering aims to discover natural instance-groupings within a given dataset. Learning cluster-based classification systems involves partitioning a training set into data subsets (clusters) and building a local classification model for each data cluster. The class of a new instance is predicted by first assigning the instance to its nearest cluster and then using that cluster’s local classification model to predict the instance’s class. In this paper, we present an ant colony optimization (ACO) approach to building cluster-based classification systems. Our ACO approach optimizes the number of clusters, the positioning of the clusters, and the choice of classification algorithm to use as the local classifier for each cluster. We also present an ensemble approach that allows the system to decide on the class of a given instance by considering the predictions of all local classifiers, employing a weighted voting mechanism based on the fuzzy degree of membership in each cluster. Our experimental evaluation employs five widely used classification algorithms: naïve Bayes, nearest neighbour, Ripper, C4.5, and support vector machines, and results are reported on a suite of 54 popular UCI benchmark datasets.

Keywords

Ant colony optimization (ACO) Data mining Classification Mixture model Ensemble methods 

Notes

Acknowledgements

The partial support of a grant from the Brandon University Research Council (BURC) is gratefully acknowledged. The authors would like to thank the anonymous reviewers and the guest editor for their insightful comments which substantially improved the paper.

Supplementary material

11721_2017_138_MOESM1_ESM.pdf (69 kb)
Supplementary material 1 (pdf 68 KB)
11721_2017_138_MOESM2_ESM.rar (10.7 mb)
Supplementary material 2 (rar 10945 KB)
11721_2017_138_MOESM3_ESM.rar (12.4 mb)
Supplementary material 3 (rar 12678 KB)

References

  1. Abdallah, Z. S., Gaber, M., Srinivasan, B., & Krishnaswamy, S. (2012). CBARS: Cluster based classification for activity recognition systems. In A. Hassanien, A. Salem, R. Ramadan & T. Kim (Eds.), Advanced machine learning technologies and applications (Proceedings AMLTA-2012), Communications in Computer and Information Science (CCIS) (Vol. 322, pp. 82–91). Berlin: Springer.Google Scholar
  2. Abdelbar, A. M., & Salama, K. M. (2016). Clustering with the ACO\({_{\mathbb{R}}}\) algorithm. In: M. Dorigo, M. Birattari, X. Li, M. López-Ibáñez, K. Ohkura, C. Pinciroli & T. Stützle (Eds.), Swarm intelligence (Proceedings ANTS-2016), Lecture Notes in Computer Science (Vol. 9882, pp. 291–292). Berlin: Springer.Google Scholar
  3. Arthur, D., & Vassilvitskii, S. (2007). k-means\(++\): The advantages of careful seeding. In Proceedings ACM-SIAM Symposium on Discrete Algorithms (SODA-2007) (pp. 1027–1035). Philadelphia, PA: Society for Industrial and Applied Mathematics.Google Scholar
  4. Ashok, L., & Messinger, D. (2012). A spectral image clustering algorithm based on ant colony optimization. In Proceedings SPIE 8390, Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XVIII (p. 83901P). SPIE Press.Google Scholar
  5. Asuncion, A., & Newman, D. (2007). UCI Machine Learning Repository. http://www.ics.uci.edu/~mlearn/MLRepository.html.
  6. Bezdek, J., Ehrlich, R., & Full, W. (1984). FCM: The fuzzy \(c\)-means clustering algorithm. Computers & Geosciences, 10(2–3), 191–203.CrossRefGoogle Scholar
  7. Bishop, C. M. (2007). Pattern recognition and machine learning. Berlin: Springer.MATHGoogle Scholar
  8. Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140.MATHGoogle Scholar
  9. Dempster, A., Laird, N., & Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B (Methodological), 39(1), 1–8.MathSciNetMATHGoogle Scholar
  10. Dorigo, M., & Stützle, T. (2004). Ant colony optimization. Cambridge: MIT Press.MATHGoogle Scholar
  11. Franca, F., Coelho, G., & Zuben, F. (2008). bicACO: An ant colony inspired biclustering algorithm. In M. Dorigo, M. Birattari, C. Blum, M. Clerc, T. Stützle & A. Winfield (Eds.), Ant colony optimization and swarm intelligence (Proceedings ANTS-2008), Lecture Notes in Computer Science (Vol. 5217, pp. 401–402). Berlin: Springer.Google Scholar
  12. Freund, Y., & Schapire, R. (1997). Decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1), 119–139.MathSciNetCrossRefMATHGoogle Scholar
  13. Gan, G., Ma, C., & Wu, J. (2007). Data clustering: Theory, algorithms, and applications. Philadelphia, PA: SIAM Press.CrossRefMATHGoogle Scholar
  14. Han, J., & Kamber, M. (2000). Data mining: Concepts and techniques. San Francisco, CA: Morgan Kaufmann.MATHGoogle Scholar
  15. Jafar, M., & Sivakumar, R. (2010). Ant-based clustering algorithms: A brief survey. International Journal of Computer Theory and Engineering, 2, 787–796.CrossRefGoogle Scholar
  16. Kao, Y., & Cheng, K. (2016). An ACO-based clustering algorithm. In: M. Dorigo, L. M. Gambardella, M. Birattari, A. Martinoli, R. Poli & T. Stützle (Eds.), Swarm intelligence (Proceedings ANTS-2006), Lecture Notes in Computer Science (Vol. 4150, pp 210–221). Berlin: Springer.Google Scholar
  17. Kaufman, L., & Rousseeuw, P. (1987). Clustering by means of medoids. In Y. Dodge (Ed.), Statistical data analysis based on the L1 norm and related methods (pp. 405–416). Amsterdam: North-Holland.Google Scholar
  18. Liao, T., Socha, K., Montes de Oca, M., Stützle, T., & Dorigo, M. (2014). Ant colony optimization for mixed-variable optimization problems. IEEE Transactions on Evolutionary Computation, 18(4), 503–518.CrossRefGoogle Scholar
  19. Liu, X. Y., & Fu, H. (2010). An effective clustering algorithm with ant colony. Journal of Computers, 5, 598–605.Google Scholar
  20. Liu, Y., Li, Z., Xiong, H., Gao, X., & Wu, J. (2010). Understanding of internal clustering validation measures. In Proceedings IEEE conference on data mining (pp. 911–916). IEEE Press.Google Scholar
  21. MacQueen, J. (1967). Some methods of classification and analysis of multivariate observations. In Proceedings fifth Berkeley symposium on mathematical statistics and probability (Vol. 1, pp. 281–297). University of California Press.Google Scholar
  22. Martens, D., De Backer, M., Haesen, R., Vanthienen, J., Snoeck, M., & Baesens, B. (2007). Classification with ant colony optimization. IEEE Transactions on Evolutionary Computation, 11(5), 651–665.CrossRefGoogle Scholar
  23. Mehrotra, K., Ozgencil, N., & McCracken, N. (2007). Squeezing the last drop: Cluster-based classification algorithm. Statistics & Probability Letters, 77, 1288–1299.MathSciNetCrossRefMATHGoogle Scholar
  24. Menendez, H., Otero, F., & Camacho, D.(2014a). MACOC: A medoid-based ACO clustering algorithm. In: M. Dorigo, M. Birattari, S. Garnier, H. Hamann, M. Montes de Oca, C. Solnon & T. Stützle (Eds.,) Swarm Intelligence (Proceedings ANTS-2014), Lecture Notes in Computer Science (Vol. 8667, pp. 122–133). Berlin: Springer.Google Scholar
  25. Menendez, H., Otero, F., & Camacho, D. (2014b). SACOC: A spectral-based ACO clustering algorithm. In Proceedings eighth international symposium on intelligent distributed computing, Studies in Computational Intelligence (Vol. 570, pp. 185–194). Berlin: SpringerGoogle Scholar
  26. Otero, F. E., Freitas, A. A., & Johnson, C. G. (2009). Handling continuous attributes in ant colony classification algorithms. In IEEE Symposium on Computational Intelligence in Data Mining (CIDM 2009) (pp 225–231). IEEE Press.Google Scholar
  27. Otero, F. E., Freitas, A. A., & Johnson, C. G. (2012). Inducing decision trees with an ant colony optimization algorithm. Applied Soft Computing, 12(11), 3615–3626.CrossRefGoogle Scholar
  28. Otero, F. E., Freitas, A. A., & Johnson, C. (2013). A new sequential covering strategy for inducing classification rules with ant colony algorithms. IEEE Transactions on Evolutionary Computation, 17(1), 64–74.CrossRefGoogle Scholar
  29. Ozgencil, N. E. (2007). Cluster based classification for semantic role labeling. Ph.D. thesis, Syracuse University, Syracuse, NY, USA.Google Scholar
  30. Parpinelli, R. S., Lopes, H. S., & Freitas, A. A. (2002). Data mining with an ant colony optimization algorithm. IEEE Transactions on Evolutionary Computation, 6(4), 321–332.CrossRefMATHGoogle Scholar
  31. Rokach, L. (2010). Ensemble-based classifiers. Artificial Intelligence Review, 33(1–2), 1–39.CrossRefGoogle Scholar
  32. Salama, K. M., & Abdelbar, A. M. (2015). Learning neural network structures with ant colony algorithms. Swarm Intelligence, 9(4), 229–265.CrossRefGoogle Scholar
  33. Salama, K. M., & Abdelbar, A. M. (2016). Using ant colony optimization to build cluster-based classification systems. In M. Dorigo, M. Birattari, X. Li, M. López-Ibáñez, K. Ohkura, C. Pinciroli & T. Stützle (Eds.), Swarm Intelligence (Proceedings ANTS-2016), Lecture Notes in Computer Science (Vol. 9882, pp. 210–221). Berlin: Springer.Google Scholar
  34. Salama, K. M., & Freitas, A. A. (2013a). Clustering-based Bayesian multi-net classifier construction with ant colony optimization. In IEEE Congress on Evolutionary Computation (IEEE CEC) (pp. 3079–3086). IEEE Press.Google Scholar
  35. Salama, K. M., & Freitas, A. A. (2013b). Learning Bayesian network classifiers using ant colony optimization. Swarm Intelligence, 7(2–3), 229–254.CrossRefGoogle Scholar
  36. Salama, K. M., & Freitas, A. A. (2014a). ABC-Miner\(+\): Constructing Markov blanket classifiers with ant colony algorithms. Memetic Computing, 6(3), 183–206.CrossRefGoogle Scholar
  37. Salama, K. M., & Freitas, A. A. (2014b). Classification with cluster-based Bayesian multi-nets using ant colony optimization. Swarm and Evolutionary Computation, 18, 54–70.CrossRefGoogle Scholar
  38. Salama, K. M., & Freitas, A. A. (2015). Ant colony algorithms for constructing Bayesian multi-net classifiers. Intelligent Data Analysis, 19(2), 233–257.Google Scholar
  39. Salama, K. M., & Otero, F. E. (2014). Learning multi-tree classification models with ant colony optimization. In Proceedings sixth international conference on evolutionary computation theory and applications (ECTA-2014), Lecture Notes in Computer Science (pp. 38–48). Berlin: Springer.Google Scholar
  40. Salama, K. M., Abdelbar, A. M., & Freitas, A. A. (2011). Multiple pheromone types and other extensions to the Ant-Miner classification rule discovery algorithm. Swarm Intelligence, 5(3–4), 149–182.CrossRefGoogle Scholar
  41. Salama, K. M., Abdelbar, A. M., & Anwar, I. M. (2016). Data reduction for classification with ant colony optimization. Intelligent Data Analysis, 20(5), 1021–1059.CrossRefGoogle Scholar
  42. Salama, K. M., Abdelbar, A. M., Helal, A. Z., & Freitas, A. A. (2017). Instance-based classification with ant colony optimization. Intelligent Data Analysis (to appear).Google Scholar
  43. Schapire, R. (1990). The strength of weak learnability. Machine Learning, 5(2), 197–227.Google Scholar
  44. Shelokar, P. S., Jayaraman, V. K., & Kulkarni, B. D. (2004). An ant colony approach for clustering. Analytica Chimica Acta, 509(2), 187–195.CrossRefGoogle Scholar
  45. Socha, K., & Blum, C. (2007). An ant colony optimization algorithm for continuous optimization: Application to feed-forward neural network training. Neural Computing & Applications, 16, 235–247.CrossRefGoogle Scholar
  46. Socha, K., & Dorigo, M. (2008). Ant colony optimization for continuous domains. European Journal of Operational Research, 185, 1155–1173.MathSciNetCrossRefMATHGoogle Scholar
  47. Tan, P. N., Steinbach, M., & Kumar, V. (2005). Introduction to data mining. Reading: Addison Wesley.Google Scholar
  48. Witten, I. H., & Frank, E. (2010). Data mining: Practical machine learning tools and techniques. San Francisco, CA: Morgan Kaufmann.MATHGoogle Scholar
  49. Woniak, M., Graña, M., & Corchado, E. (2014). A survey of multiple classifier systems as hybrid systems. Information Fusion, 16, 3–17.CrossRefGoogle Scholar
  50. Xu, R., & Wunsch, D. (2009). Clustering. Piscataway, NJ: IEEE Press.Google Scholar
  51. Yen, S. J., & Lee, Y. S. (2009). Cluster-based under-sampling approaches for imbalanced data distributions. Expert Systems with Applications, 36, 5718–5727.CrossRefGoogle Scholar
  52. Zhang, X., & Xiao, W. (2012). Clustering based two-stage text classification requiring minimal training data. Computer Science and Information Systems, 9(4), 1627–1643.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.School of ComputingUniversity of KentCanterburyUK
  2. 2.Department of Mathematics and Computer ScienceBrandon UniversityBrandonCanada

Personalised recommendations