Skip to main content
SpringerLink
Log in

Neuron Pruning-Based Discriminative Extreme Learning Machine for Pattern Classification

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Extreme learning machine (ELM), as a newly developed learning paradigm for the generalized single hidden layer feedforward neural networks, has been widely studied due to its unique characteristics, i.e., fast training, good generalization, and universal approximation/classification ability. A novel framework of discriminative extreme learning machine (DELM) is developed for pattern classification. In DELM, the margins between different classes are enlarged as much as possible through a technique called ε-dragging. DELM is further extended to pruning DELM (P-DELM) using L2,1-norm regularization. The performance of DELM is compared with several state-of-the-art methods on public face databases. The simulation results show the effectiveness of DELM for face recognition when there are posture, facial expression, and illumination variations. P-DELM can distinguish the importance of different hidden neurons and remove the worthless ones. The model can achieve promising performance with fewer hidden neurons and less prediction time on several benchmark datasets. In DELM model, the margins between different classes are enlarged by learning a nonnegative label relaxation matrix. The experiments validate the effectiveness of DELM. Furthermore, DELM is extended to P-DELM based on L2,1-norm regularization. The developed P-DELM can naturally distinguish the importance of different hidden neurons, which will lead to a more compact network by neuron pruning. Experimental validations on some benchmark datasets show the advantages of the proposed P-DELM method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Taylor JG. Cognitive computation. Cogn Comput. 2009;1(1):4–16.

    Article  Google Scholar 

  2. Clark A. Mindware: an introduction to the philosophy of cognitive science. New York: Oxford University Press; 2001.

    Google Scholar 

  3. Luo B, Hussain A, Mahmud M, et al. Advances in brain-inspired cognitive systems. Cogn Comput. 2016;8(5):795–6.

    Google Scholar 

  4. Zhang HY, Ji P, Wang JQ, et al. A neutrosophic normal cloud and its application in decision-making. Cogn Comput. 2016;8(4):1–21.

    Article  Google Scholar 

  5. Gepperth A, Karaoguz C. A bio-inspired incremental learning architecture for applied perceptual problems. Cogn Comput. 2016;8(5):924–34.

    Article  Google Scholar 

  6. Rumelhart DE, Hinton GE, Williams RJ. Learning representation by backpropagating errors. Nature. 1986;323(6088):533–6.

    Article  Google Scholar 

  7. Zhang L, Zhang D, Tian F. SVM and ELM: who wins? Object recognition with deep convolutional features from ImageNet. Comput Sci. 2015.

  8. Huang G-B, Zhou H, Ding X, Zhang R. Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man CybernB Cybern. 2012;42(2):513–29.

    Article  Google Scholar 

  9. Huang GB. What are extreme learning machines? Filling the gap between Frank Rosenblatt’s dream and John von Neumann’s puzzle. Cogn Comput. 2015;7(3):263–78.

    Article  Google Scholar 

  10. Huang GB, Chen L, Siew CK. Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw. 2006;17(4):879–92.

    Article  PubMed  Google Scholar 

  11. Huang GB, Li MB, Chen L, Siew CK. Incremental extreme learning machine with fully complex hidden nodes. Neurocomputing. 2008;71(4–6):576–83.

    Article  Google Scholar 

  12. Huang GB. An insight into extreme learning machines: random neurons, random features and kernels. Cogn Comput. 2014;6(3):376–90.

    Article  Google Scholar 

  13. Fusi S, Miller EK, Rigotti M. Why neurons mix: high dimensionality for higher cognition. Curr Opin Neurobiol. 2016;37:66.

    Article  CAS  PubMed  Google Scholar 

  14. Huang G, Song S, Gupta JND, et al. Semi-supervised and unsupervised extreme learning machines. IEEE Trans Cybern. 2014;44(12):2405–17.

    Article  PubMed  Google Scholar 

  15. Tang J, Deng C, Huang GB. Extreme learning machine for multilayer perceptron. IEEE Trans Neural Netw Learn Syst. 2016;27(4):809–21.

    Article  PubMed  Google Scholar 

  16. Lekamalage CL, Yang Y, Huang GB, et al. Dimension reduction with extreme learning machine. IEEE Trans Image Process. 2016;25(8):3906–18.

    Article  Google Scholar 

  17. Liu H, Sun F, Yu Y. Multitask extreme learning machine for visual tracking. Cogn Comput. 2014;6(3):391–404.

    Article  Google Scholar 

  18. Liu H, Qin J, Sun F, et al. Extreme kernel sparse learning for tactile object recognition. IEEE Trans Cybern, 2016.

  19. Zhang L, Zhang D. Domain adaptation extreme learning machines for drift compensation in E-nose systems. IEEE Trans Instrum Meas. 2015;64:1790–801.

    Article  Google Scholar 

  20. Zhang L, Zhang D. Robust visual knowledge transfer via extreme learning machine based domain adaptation. IEEE Trans Image Process. 2016;25(10):4959–73.

    Article  Google Scholar 

  21. Rong HJ, Ong YS, Tan AH, et al. A fast pruned-extreme learning machine for classification problem. Neurocomputing. 2008;72(1–3):359–66.

    Article  Google Scholar 

  22. Lan Y, Soh YC, Huang GB. Constructive hidden nodes selection of extreme learning machine for regression. Neurocomputing. 2010;73(16–18):3191–9.

    Article  Google Scholar 

  23. Miche Y, Sorjamaa A, Bas P, et al. OP-ELM: optimally pruned extreme learning machine. IEEE Trans Neural Netw. 2010;21(1):158–62.

    Article  PubMed  Google Scholar 

  24. He B, Sun T, Yan T, et al. A pruning ensemble model of extreme learning machine with L1/2 regularizer. Multidimension Syst Signal Process. 2016:1–19.

  25. Xiang S, Nie F, Meng G, et al. Discriminative least squares regression for multiclass classification and feature selection. IEEE Trans Neural Netw Learn Syst. 2012;23(11):1738–54.

    Article  PubMed  Google Scholar 

  26. Liu H. Robust exemplar extraction using structured sparse coding. IEEE Trans Neural Netw Learn Syst. 2015;26(8):1816–21.

    Article  PubMed  Google Scholar 

  27. Liu H, Guo D, Sun F. Object recognition using tactile measurements: kernel sparse coding methods. IEEE Trans Instrum Meas. 2016;65(3):656–65.

    Article  Google Scholar 

  28. Nie F, Huang H, Cai X, et al. Efficient and robust feature selection via joint ℓ2, 1-norms minimization. Adv Neural Info Process Syst. 2010:1813–1821.

  29. Zhao W, Chellappa R, Phillips J, Rosenfeld A. Face recognition: a literature survey. ACM Comput Surv. 2003;35(4):399–458.

    Article  Google Scholar 

  30. Samaria F S, Harter A C. Harter, A.: Parameterisation of a stochastic model for human face identification[C]// Applications of computer vision, 1994. Proceedings of the Second IEEE Workshop on. 1995:138–142.

  31. Rizon M, Hashim MF, Saad P, et al. Face recognition using eigenfaces and neural networks. Am J Appl Sci. 2006;3(6):586–91.

    Google Scholar 

  32. Georghiades AS, Belhumeur PN, Kriegman DJ. From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Trans Pattern Anal Mach Intell. 2001;23(6):643–60.

    Article  Google Scholar 

  33. Martinez, A. M. The AR face database. Cvc Technical Report, 2010, 24.

  34. Vapnik V. Statistical learning theory. New York: Wiley; 1998.

    Google Scholar 

  35. Suykens JAK, Vandewalle J. Least squares support vector machine classifiers. Neural Process Lett. 1999;9(3):293–300.

    Article  Google Scholar 

  36. M. Turk, A. Pentland, Face recognition using eigenfaces. In: Proceeding of the CVPR, 1991.

  37. C. Blake, C. Merz, UCI repository of machine learning databases [online], in (http://www.ics.uci.edu/mlearn/MLRepository.html), Department of Information and Computer Sciences, University of California, Irvine, USA, 1998.

  38. S. A. Nene, S. K. Nayar, H. Murase, et al., Columbia object image library (COIL-20), Technical report, technical report CUCS-005-96, 1996.

  39. Yu J, Tao D, Wang M, Yu J, Tao D, Wang M. Adaptive hypergraph learning and its application in image classification. IEEE Trans Image Process. 2012;21(7):3262–72.

    Article  PubMed  Google Scholar 

  40. G. Griffin, A. Holub, P. Perona, Caltech-256 object category dataset, Technical Report, California Institute of Technology, 2007.

  41. Bergamo A, Torresani L, Fitzgibbon A. PICODES: learning a compact code for novel-category recognition. Adv Neural Info Process Syst. 2011:2088–2096.

  42. Liu H, Yu Y, Sun F, et al. Visual-tactile fusion for object recognition. IEEE Trans Automation Sci Eng. 2016:1–13.

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 61571069, 61401048), Chongqing University Postgraduates’ Innovation Project (No. CYB15030), and in part by the Fundamental Research Funds for the Central Universities.

Author information

Authors and Affiliations

  1. College of Communication Engineering, Chongqing University, Chongqing, 400044, China

    Tan Guo, Lei Zhang & Xiaoheng Tan

Authors
  1. Tan Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoheng Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoheng Tan.

Ethics declarations

Funding

This study was funded by the National Natural Science Foundation of China (grant number 61571069 and 61401048), Chongqing University Postgraduates’ Innovation Project (grant number CYB15030), and in part by the Fundamental Research Funds for the Central Universities.

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with animals performed by any of the authors.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, T., Zhang, L. & Tan, X. Neuron Pruning-Based Discriminative Extreme Learning Machine for Pattern Classification. Cogn Comput 9, 581–595 (2017). https://doi.org/10.1007/s12559-017-9474-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-017-9474-4

Keywords

Search

Navigation