Skip to main content
SpringerLink
Log in

Particle Swarm Optimization Based Swarm Intelligence for Active Learning Improvement: Application on Medical Data Classification

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Semi-supervised learning targets the common situation where labeled data are scarce but unlabeled data are abundant. It uses unlabeled data to help supervised learning tasks. In practice, it may make sense to utilize active learning in conjunction with semi-supervised learning. That is, we might allow the learning algorithm to pick a set of unlabeled instances to be labeled by a domain expert, which will then be used as the labeled data set. However, existing approaches are computationally expensive and require searching through an entire unlabeled dataset, which may contain redundant instances that provide no instructive information to the classifier and can decrease the performance. To address this optimization problem, a hybrid system that combines active learning (AL) and particle swarm optimization (PSO) algorithms is proposed to reduce the cost of labeling while building a more efficient classifier. The novelty of this work resides in the integration of a bio-inspired optimization algorithm in the machine learning strategy. Furthermore, a novel uncertainty measure was integrated into the particle swarm optimization algorithm as an objective function to select from massive amounts of medical instances those that are deemed most informative. To evaluate the effectiveness of the proposed approach, eighteen (18) benchmark datasets were used and compared against three best-known classifiers with different learning paradigms: AL–NB an active learning algorithm using Naïve Base classifier and Margin Sampling strategy, SVM (Support Vector Machine), ELM (Extreme Learning Machine) with supervised learning, and TSVM (Transductive Support Vector Machine) with the semi-supervised learning. Experiments showed that the proposed approach is effective in reducing the efforts required by experts for medical data annotation to produce an accurate classifier. The active learning approach has been utilized to optimize the expensive task of labeling. Based on a novel uncertainty measure, the nature-inspired algorithm PSO attempts to select from massive amounts of unlabeled medical instances those considered informative, at the same time improving the classifier performance. The experiments carried out confirm that the proposed strategy significantly enhances the performance of the AL algorithm compared with the commonly used uncertainty strategies. It achieves a performance similar to that of fully supervised and semi-supervised algorithms while requiring much less labeling. As a future extension of this work, it would be interesting to integrate other evolutionary optimization algorithms and compare them with our approach. In addition, it is beneficial to test the impact of using other variants of PSO algorithm in our approach. Also, it is aimed to test more classification algorithms in the experimentation process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Carrizos E, Morales DR. Supervised classification and mathematical optimization. Comput Oper Res. 2013;40(1):150–65.

    MathSciNet  MATH  Google Scholar 

  2. Ahn E, Kumar A, Fulham M, Feng D, Kim J. Convolutional sparse kernel network for unsupervised medical image analysis. Med Image Anal. 2019;56:140–151.

  3. Zemmal N, Azizi N, Sellami M. CAD system for classification of mammographic abnormalities using transductive semi supervised learning algorithm and heterogeneous features; 12th International Symposium on Programming and Systems, ISPS. Algeria: Algiers; 2015. p. 245–53.

    Google Scholar 

  4. Chen S, Wang Y, Lin CT, Ding W, Cao Z. Semi-supervised feature learning for improving writer identification. Inform Sciences. 2019;482:156–170.

  5. Zemmal N, Azizi N, Ziani A, Benzebouchi NE, Aldwairi M. An enhanced feature selection approach based on mutual information for breast cancer diagnosis. 2019 6th International Conference on Image and Signal Processing and their Applications (ISPA), Mostaganem, Algeria; 2019. p. 1–6. https://doi.org/10.1109/ISPA48434.2019.89.

  6. Zhang C, Wang S, Li D, Yang J, Zhang J. Semi-supervised behavioral learning and its application. Optik - International Journal for Light and Electron Optics. 2017;127(1):376–382.

  7. Settles B. Active learning literature survey. Technical report 1648, University of Wisconsin, Madison. 2010.

  8. Wang M, Min F, Zhang ZH, Wu YX. Active learning through density clustering. Expert Syst Appl. 2017;85:305–17.

    Google Scholar 

  9. Kholghi, M., Sitbon, L., Zuccon ,G., Nguyen, A., 2017. Active Learning Reduces Annotation Time for Clinical Concept Extraction. Int J Med Inform 106, 25-31.

  10. Hoi SCH, Jin R, Zhu J, Lyu M. Batch Mode Active Learning and its Applications to Medical Image Classification. Pittsburgh, Pennsylvania: Proceedings of the 23rd International Conference on Machine Learning; 2006. p. 417–27.

    Google Scholar 

  11. Azizi N, Zemmal N, Sellami M, Farah N. A new hybrid method combining genetic algorithm and support vector machine classifier: Application to CAD system for mammogram images., International Conference on Multimedia Computing and Systems (ICMCS). Morocco: Marrekech; 2014. p. 415–20.

    Google Scholar 

  12. Phill K, Enkhbayar E, Shin DK, Minhaz UA, Songguo J. Active and semi-supervised learning for object detection with imperfect data. Cogn Syst Res. 2017;45:109–23.

    Google Scholar 

  13. Mohsen M, Karan S, Deborah F, Ian F, Javier RM, Garrison WC. Deep active object recognition by joint label and action prediction. Comput Vis Image Underst. 2017;157:128–37.

    Google Scholar 

  14. Rong H, Brian MN, Sarah JD. Active learning for text classification with reusability. Expert Syst Appl. 2017;45:438–49.

    Google Scholar 

  15. Yukun C, Subramani M, Hua X. Applying active learning to assertion classification of concepts in clinical text. J Biomed Inform. 2012;45(2):265–72.

    Google Scholar 

  16. AlRahhal MM, Yakoub B, Haikel AH, Naif A, Farid M, Yager RR. Deep learning approach for active classification of electrocardiogram signals. Inf Sci. 2016;345:340–54.

    Google Scholar 

  17. Chen Y, Carroll RJ, Hinz ERM, Shah A, Eyler AE, Denny JC, et al. Applying active learning to high-throughput phenotyping algorithms for electronic health records data. J Am Med Inform Assoc. 2013;20(2):253–9.

    Google Scholar 

  18. Jiaji H, Rewon C, Vinay R, Hairong L, Sanjeev S, Adam C. Active Learning for Speech Recognition: The Power of Gradients. The 30th Conference on Neural Information Processing Systems, NIPS. Barcelona, Spain; 2016. p. 1–5.

  19. Yu D, Varadarajan B, Deng L. Active learning and semi-supervised learning for speech recognition: a unified framework using the global entropy reduction maximization criterion. Comput Speech Lang. 2010;24(3):433–44.

    Google Scholar 

  20. Yi Y, Zhigang M, Feiping N, Xiaojun C, Alexander GH. Multi-Class Active Learning by Uncertainty Sampling with Diversity Maximization. Int J Comput Vis. 2015;113(2):113–27.

    MathSciNet  Google Scholar 

  21. Yifan F, Xingquan Z, Bin L. A survey on instance selection for active learning. Knowl Inf Syst. 2013;35(2):249–83.

    Google Scholar 

  22. Christensen J, Bastien C. Heuristic and meta-heuristic optimization algorithms. In: Christensen J, Bastien C (eds) Nonlinear optimization of vehicle safety structures. Butterworth-Heinemann, Oxford; 2016. p 277–314

  23. Amir-Reza A, Afsane B. A novel hybrid meta-heuristic technique applied to the well-known benchmark optimization problems. J Ind Eng Int. 2017;13(1):93–105.

    Google Scholar 

  24. Kennedy J, Eberhart R. Particle swarm optimization. Proceedings of ICNN'95 - International Conference on Neural Networks, Perth, WA, Australia, 1995, p. 1942-1948 vol.4. https://doi.org/10.1109/ICNN.1995.488968.

  25. Aldwairi M, Duwairi R, Alqarqaz W. A Classification System for Predicting RNA Hairpin Loops. International Joint Conference on Bioinformatics, Systems Biology and Intelligent Computing. Shanghai. 2009, p. 109-115. https://doi.org/10.1109/IJCBS.2009.123.

  26. Salim L. Glioma detection based on multi-fractal features of segmented brain MRI by particle swarm optimization techniques. Biomed Signal Proces. 2017;31:148–55.

    Google Scholar 

  27. Aldwairi M, Alsalman R. MALURLs: Malicious URLs Classification System. The Annual International Conference on Information Theory and Applications, Singapore, Feb 28, 2011; 2011. https://doi.org/10.5176/978-981-08-8113-9_ITA2011-29.

  28. Parham M, Mozhgan G. A hybrid particle swarm optimization for feature subset selection by integrating a novel local search strategy. Appl Soft Comput. 2016;43:117–30.

    Google Scholar 

  29. Kunjie Y, Xin W, Zhenlei W. Multiple learning particle swarm optimization with space transformation perturbation and its application in ethylene cracking furnace optimization. Knowl-Based Syst. 2016;96:156–70.

    Google Scholar 

  30. Aldwairi M, Khamayseh Y, Al-Masri M. Application of Artificial Bee Colony for Intrusion Detection Systems, Security and Communication Networks, vol. 8(16): John Wiley & Sons, Ltd.; 2015. p. 2730–40. 2015/11

  31. Chang PC, Lin J, Liu CH. An attribute weight assignment and particle swarm optimization algorithm for medical database classifications. Comput Methods Programs Biomed. 2012;107(3):382–392.

  32. Fernández A, Sara DR, Nitesh VC, Francisco H. An insight into imbalanced Big Data classification: outcomes and challenges. Complex Intell Syst. 2017; 3. 105–120.

  33. Benzebouchi NE, Azizi N, Ashour AS, Dey N, Sherratt RS. Multi-modal classifier fusion with feature cooperation for glaucoma diagnosis. J Exp Theor Artif In. 2019;31(6):841–874.

  34. Shashi P, Parag N, Vipan K, Amod K. PSO Aided Adaptive Complementary Filter for Attitude Estimation. J Intell Robot Syst. 2017;87(3-4):531–43.

    Google Scholar 

  35. Siamak SG, Naman G, Bistra D. Active Learning in Multi-Objective Evolutionary Algorithms for Sustainable Building Design. The Genetic and Evolutionary Computation Conference; 2016. p. 589–96. https://doi.org/10.1145/2908812.2908947

  36. Junio F, Gisele LP, Altigran SS, Marcos AG, Edleno M, Adriano V, et al. Active Learning Genetic programming for record deduplication. IEEE Trans Knowl Data Eng. 2012;24(3):399–412.

    Google Scholar 

  37. Axel-Cyrille NN, Klaus L. EAGLE: Efficient Active Learning of Link Specifications Using Genetic Programming. In: Proceeding of Extended Semantic Web Conference: The Semantic Web: Research and Applications; 2012. p. 149–63.

    Google Scholar 

  38. Begüm D, Lorenzo B. A Novel Active Learning Method in Relevance Feedback for Content-Based Remote Sensing Image Retrieval. IEEE Trans Geosci Remote Sens. 2015;53(5):2323–34.

    Google Scholar 

  39. Handing W, Yaochu J, John D. Committee-based Active Learning for Surrogate-Assisted Particle Swarm Optimization of Expensive Problems. IEEE Trans Cybernet. 2017;47(9):2664–77.

    Google Scholar 

  40. D. Gorisse, M. Cord and F. Precioso. Optimization on active learning strategy for object category retrieval. 16th IEEE International Conference on Image Processing, Cairo, 2009. p. 1873–6. https://doi.org/10.1109/ICIP.2009.5413554 

  41. Guoliang H, Yifei L, Wen Z. An uncertainty and density based active semi-supervised learning scheme for positive unlabeled multivariate time series classification. Knowl-Based Syst. 2017;124:80–92.

    Google Scholar 

  42. Hualong Y, Changyin S, Wankou Y, Xibei Y, Xin Z. AL-ELM: One uncertainty-based active learning algorithm using extreme learning machine. Neurocomputing. 2015;166:140–50.

    Google Scholar 

  43. Benala TR, Mall R, Dehuri S, Swetha P. Software Effort Estimation Using Functional Link Neural Networks Tuned with Active Learning and Optimized with Particle Swarm Optimization. In: Panigrahi B., Suganthan P., Das S. (eds) Swarm, Evolutionary, and Memetic Computing. SEMCCO 2014. Lecture Notes in Computer Science, vol 8947. Springer, Cham; 2015. p. 223–38.

  44. Ma C, Dai Q, Liu SB. A hybrid PSO and active learning SVM model for relevance feedback in the Content-based images retrieval. In: Proceeding of The IEEE International Conference on Computer Science and Service System; 2012. p. 130–3.

    Google Scholar 

  45. Seung HS, Opper M, Sompolinsky H. Query by committee. Workshop on Computational Learning Theory, Pittsburgh Pennsylvania USA, July; 1992. p. 287–94.

  46. Lindenbaum M, Markovitch S, Rusakov D. Selective sampling for nearest neighbor classifiers. Mach Learn. 2004;54:125–52.

    MATH  Google Scholar 

  47. Scheffer T, Decomain C, Wrobel S. Active hidden Markov models for information extraction. In: Hoffmann F., Hand D.J., Adams N., Fisher D., Guimaraes G. (eds) Advances in Intelligent Data Analysis. IDA 2001. Lecture Notes in Computer Science, vol 2189. Springer, Berlin, Heidelberg; 2001. p. 309–18.

  48. Shannon CE. A mathematical theory of communication. Bell Syst Tech J. 1948;27:623–56.

    MathSciNet  MATH  Google Scholar 

  49. Harrison K, Engelbrecht AP, Ombuki-Berman MP. Inertia Control Strategies for Particle Swarm Optimization: Too Much Momentum, Not Enough Analysis. Swarm Intell. 2016;10(4):267–305.

    Google Scholar 

  50. Yang H, Xu Y, Peng G, Yu G, Chen M, Duan W. Particle swarm optimization and its application to seismic inversion of igneous rocks. Int J Min Sci Technol. 2017;27(2):349–57.

    Google Scholar 

  51. Jiaxuan W, Ruisheng Z, Zhixuan Y, Rongjing H. A BPSO-SVM algorithm based on memory renewal and enhanced mutation mechanisms for feature selection. Appl Soft Comput. 2017;58:176–92.

    Google Scholar 

  52. Alfonso MC, Ricardo BF, Herón ML, Marco-Antonio RS, Luis-Alfonso VV, Prometeo CA, et al. J Electron Test. 2017;33(4):431–47.

    Google Scholar 

  53. Sameer FO, Abu Bakar MR, Zaidan AA, Zaidan BB. A new algorithm of modified binary particle swarm optimization based on the Gustafson-Kessel for credit risk assessment. Neural Comput & Applic, 2017: 31, p. 337–46. https://doi.org/10.1007/s00521-017-3018-4.

  54. Zhong J, Zhang WN. A novel discrete particle swarm optimization to solve traveling salesman problem. IEEE Congress on Evolutionary Computation, Singapore; 2007. p. 3283–7.

  55. Jiang J, Zhai C. Instance weighting for domain adaptation in NLP. In: Annual Meeting of the Association for Computational Linguistics, Prague, Czech Republic, June; 2007. p. 254–71.

  56. Mohamed-Rafik B, Yolande B, Abdel B. An adaptive streaming active learning strategy based on instance weighting. Pattern Recogn Lett. 2016;70:38–44.

    Google Scholar 

  57. Bache K, Lichman M. UCI Machine Learning Repository. [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science; 2013.

    Google Scholar 

  58. Fumero F, Alayon S, Sanchez J, Sigut J, Gonzalez-Hernandez M. Rim-one. An open retinal image database for optic nerve evaluation. The 24th international symposium on computer-based medical systems,  Bristol; 2011. p. 1–6. Available at: http://medimrg.webs.ull.es/research/downloads/.

  59. Heath M, Bowyer K, Kopans D, Moore R, Kegelmeyer P. The digital database for screening mammography, IWDM-20005th International Workshop on Digital Mammography. Madison, WI: Medical Physics Publishing; 2001. p. 457–60. Available at: http://marathon.csee.usfedu/Mammography/Database.html.

  60. Frank E, Harrell JR. Vanderbilt Biostatistics. Department of Biostatistics, University of Vanderbilt; 2016. Available at: http://biostat.mc.vanderbilt.edu/DataSets.

  61. Weiss SM, Kapouleas I. An empirical comparison of pattern recognition, neural nets and machine learning classification methods. In: Shavlik JW, Dietterich TG, editors. Readings in Machine Learning. CA: Morgan Kauffman Publ; 1990. [https://www.fizyka.umk.pl/~duch/projects/projects/datasets.html].

    Google Scholar 

  62. Kuncheva L. Ludmila kuncheva collection, &lt. 2004. [http:// pages.bangor.ac.uk/~ mas00a /activities /real_data.html&gt]

  63. Karlijn JS, Vianda SS, Johannes BR, Friedo WD, Carmine Z, Kitty JJ. Diagnostic methods I: sensitivity, specificity, and other measures of accuracy. Kidney Int. 2009;75(12):1257–63.

    Google Scholar 

  64. Gaoxia J, Wenjian W. Error estimation based on variance analysis of k-fold cross-validation. Pattern Recogn. 2017;69:94–106.

    MATH  Google Scholar 

  65. Christopher JCB. A Tutorial on Support Vector Machines for Pattern Recognition. Data Min Knowl Disc. 1998;2:121–67.

    Google Scholar 

  66. Guang-Bin H, Qin-Yu Z, Chee-Kheong S. Extreme learning machine: theory and applications. Neurocomputing. 2006;70:489–501.

    Google Scholar 

  67. Zemmal N, Azizi N, Sellami M, Zenakhra D, Cheriguene S. Robust feature selection algorithm based on transductive SVM wrapper and genetic algorithm: application on computer-aided glaucoma classification. Int. J. Intelligent Systems Technologies and Applications 2018;17(3):310–346.

  68. Roy N, McCallum M. Toward optimal active learning through sampling estimation of error reduction. ICML '01: Proceedings of the Eighteenth International Conference on Machine Learning,  Williams College, Williamstown, MA, USA, June 28 - July 1; 2001. p. 441–8.

  69. Tong S, Koller D. Support vector machine active learning with applications to text classification. International Conference on Machine Learning,  Stanford, California, June; 2000. p. 999–1006.

Download references

Acknowledgments

This work is supported under the auspices of the General Directorate for Scientific Research and Technological Development.

Funding

This study was not funded by any organization or agency.

Author information

Authors and Affiliations

  1. Labged Laboratory, Computer Science Department, Badji Mokhtar University, Annaba, Algeria

    Nawel Zemmal, Nabiha Azizi, Mokhtar Sellami, Soraya Cheriguene & Nadjette Dendani

  2. Department of Mathematics and Computer Science, Mohamed Cherif Messaadia University, 41000, Souk Ahras, Algeria

    Nawel Zemmal

  3. Computer Science Department, Badji Mokhtar University, Annaba, Algeria

    Amel Ziani

  4. College of Technological Innovation, Zayed University, P.O. Box 144534, Abu Dhabi, United Arab Emirates

    Monther AlDwairi

Authors
  1. Nawel Zemmal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nabiha Azizi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mokhtar Sellami
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Soraya Cheriguene
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Amel Ziani
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Monther AlDwairi
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Nadjette Dendani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nawel Zemmal.

Ethics declarations

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Conflict of interest

The authors declare that they have no conflict of interest.

.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zemmal, N., Azizi, N., Sellami, M. et al. Particle Swarm Optimization Based Swarm Intelligence for Active Learning Improvement: Application on Medical Data Classification. Cogn Comput 12, 991–1010 (2020). https://doi.org/10.1007/s12559-020-09739-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-020-09739-z

Keywords

Search

Navigation