Skip to main content
SpringerLink
Log in

Hessian unsupervised extreme learning machine

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Extreme learning machines (ELMs) are shown to be efficient and effective learning algorithms for regression and classification tasks. ELMs, however, are typically utilized to solve supervised learning problems. Only a handful of research on ELMs focuses on exploring unlabeled data. One representative work is the unsupervised extreme learning machine (US-ELM), where the standard ELM is expanded for unsupervised learning based on Laplacian regularization. However, Laplacian regularization has poor extrapolation power since it tends to bias the solution towards a constant function. In this paper, we propose a new framework termed Hessian unsupervised ELM (HUS-ELM) to enhance the unsupervised learning of ELM. In particular, Hessian regularization can properly exploit the intrinsic local geometry of the data manifold compared to Laplacian regularization. This leverages the performance of HUS-ELM in unsupervised learning problems since the Hessian regularization can correctly reflect the positional relationship between the unlabeled samples. Six publicly available datasets are used to evaluate the proposed algorithm. The experimental results indicate that the proposed method performs better than other unsupervised learning methods in terms of clustering accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Data availability

No datasets were generated during the current study. All datasets used in this work are publicly available.

References

  1. Hagan MT, Menhaj MB (1994) Training feedforward networks with the marquardt algorithm. IEEE Trans Neural Network 5(6):989–993

    Article  Google Scholar 

  2. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323(6088):533–536

    Article  Google Scholar 

  3. Zhang L, Suganthan PN (2016) A comprehensive evaluation of random vector functional link networks. Information sciences 367:1094–1105

    Article  Google Scholar 

  4. Zhang P-B, Yang Z-X (2020) A new learning paradigm for random vector functional-link network: Rvfl+. Neural Networks 122:94–105

    Article  Google Scholar 

  5. Guo P, Lyu MR (2004) A pseudoinverse learning algorithm for feedforward neural networks with stacked generalization applications to software reliability growth data. Neurocomputing 56:101–121

    Article  Google Scholar 

  6. Huang G-B, Zhu Q-Y, Siew C-K (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541), vol. 2, pp. 985–9902. https://doi.org/10.1109/IJCNN.2004.1380068

  7. Huang G-B, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42(2), 513–529. https://doi.org/10.1109/TSMCB.2011.2168604

  8. Bartlett P. (1998) Thesamplecomplexityofp atternclassification withneuralnetworks: Thesizeoftheweightsismo reimportantthan thesizeofthenetwork. IEEETrans. Inf. Theory 44(2)

  9. Al-Dhief FT, Baki MM, Latiff NMA, Malik NNNA, Salim NS, Albader MAA, Mahyuddin NM, Mohammed MA (2021) Voice pathology detection and classification by adopting online sequential extreme learning machine. IEEE Access 9:77293–77306. https://doi.org/10.1109/ACCESS.2021.3082565

    Article  Google Scholar 

  10. Jamei M, Ahmadianfar I, Karbasi M, Malik A, Kisi O, Yaseen ZM (2023) Development of wavelet-based kalman online sequential extreme learning machine optimized with boruta-random forest for drought index forecasting. Eng Appl Artificial Intellig 117:105545

    Article  Google Scholar 

  11. Zhou X, Huang J, Lu F, Zhou W, Liu P (2023) A novel compound fault-tolerant method based on online sequential extreme learning machine with cycle reservoir for turbofan engine direct thrust control. Aeros Sci Tech 132:108059

    Article  Google Scholar 

  12. Mao W, Wang J, Xue Z (2017) An elm-based model with sparse-weighting strategy for sequential data imbalance problem. Int J Machine Learn Cyber 8:1333–1345

    Article  Google Scholar 

  13. Yu H, Yang X, Zheng S, Sun C (2019) Active learning from imbalanced data: A solution of online weighted extreme learning machine. IEEE Trans Neural Networks Learn Syst 30(4):1088–1103. https://doi.org/10.1109/TNNLS.2018.2855446

    Article  Google Scholar 

  14. Bhatia A, Chug A, Prakash Singh A (2020) Application of extreme learning machine in plant disease prediction for highly imbalanced dataset. J Statistics Manag Syst 23(6):1059–1068

    Google Scholar 

  15. He Q, Jin X, Du C, Zhuang F, Shi Z (2014) Clustering in extreme learning machine feature space. Neurocomputing 128:88–95

    Article  Google Scholar 

  16. Liu T, Lekamalage CKL, Huang G-B, Lin Z (2018) Extreme learning machine for joint embedding and clustering. Neurocomputing 277:78–88

    Article  Google Scholar 

  17. Chen J, Zeng Y, Li Y, Huang G-B (2020) Unsupervised feature selection based extreme learning machine for clustering. Neurocomputing 386:198–207

    Article  Google Scholar 

  18. Hsu Y-S, Lin S-J (2016) An emerging hybrid mechanism for information disclosure forecasting. Int J Machine Learn Cyber 7:943–952

    Article  Google Scholar 

  19. Baig MM, Awais MM, El-Alfy E-SM (2017) Adaboost-based artificial neural network learning. Neurocomputing 248:120–126

    Article  Google Scholar 

  20. Huang G, Song S, Gupta JND, Wu C (2014) Semi-supervised and unsupervised extreme learning machines. IEEE Trans Cyber 44(12):2405–2417. https://doi.org/10.1109/TCYB.2014.2307349

    Article  Google Scholar 

  21. Yang L, Yang S, Li S, Liu Z, Jiao L (2017) Incremental laplacian regularization extreme learning machine for online learning. Appl Soft Compu 59:546–555

    Article  Google Scholar 

  22. Liu M, Liu B, Zhang C, Wang W, Sun W (2017) Semi-supervised low rank kernel learning algorithm via extreme learning machine. Int J Machine Learn Cyber 8:1039–1052

    Article  Google Scholar 

  23. Yao L, Ge Z (2017) Deep learning of semisupervised process data with hierarchical extreme learning machine and soft sensor application. IEEE Trans Indust Electronics 65(2):1490–1498

    Article  Google Scholar 

  24. Gharebaghi F, Amiri A (2022) Lp-mltsvm: Laplacian multi-label twin support vector machine for semi-supervised classification. IEEE Access 10:13738–13752. https://doi.org/10.1109/ACCESS.2021.3139929

    Article  Google Scholar 

  25. Gao S, Tsang IW-H, Chia L-T (2013) Laplacian sparse coding, hypergraph laplacian sparse coding, and applications. IEEE Trans Pattern Analysis Machine Intellig 35(1):92–104. https://doi.org/10.1109/TPAMI.2012.63

    Article  Google Scholar 

  26. Krishnasamy G, Paramesran R (2019) Multiview laplacian semisupervised feature selection by leveraging shared knowledge among multiple tasks. Signal Processing: Image Communication 70:68–78. https://doi.org/10.1016/j.image.2018.09.008

    Article  Google Scholar 

  27. Kim K, Steinke F, Hein M (2009) Semi-supervised regression using hessian energy with an application to semi-supervised dimensionality reduction. In: Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C., Culotta, A. (eds.) Advances in Neural Information Processing Systems, vol. 22. Curran Associates, Inc., https://proceedings.neurips.cc/paper/2009/file/f4552671f8909587cf485ea990207f3b-Paper.pdf

  28. Krishnasamy G, Paramesran R (2016) Hessian semi-supervised extreme learning machine. Neurocomputing 207:560–567. https://doi.org/10.1016/j.neucom.2016.05.039

    Article  Google Scholar 

  29. Donoho DL, Grimes C (2003) Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proceedings of the National Academy of Sciences 100(10), 5591–5596. https://www.pnas.org/doi/pdf/10.1073/pnas.1031596100. https://doi.org/10.1073/pnas.1031596100

  30. Liu W, Ma T, Tao D, You J (2016) Hsae: A hessian regularized sparse auto-encoders. Neurocomputing 187:59–65. https://doi.org/10.1016/j.neucom.2015.07.119

    Article  Google Scholar 

  31. Pan Y, Zhou Y, Liu W, Nie L (2019) Principal component analysis on graph-hessian. In: 2019 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1494–1501. https://doi.org/10.1109/SSCI44817.2019.9002887

  32. Huang G-B, Chen L, Siew CK et al (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neural Networks 17(4):879–892

    Article  Google Scholar 

  33. Huang G-B, Chen L (2007) Convex incremental extreme learning machine. Neurocomputing 70(16–18):3056–3062

    Article  Google Scholar 

  34. Eells J, Lemaire L (1983) Selected Topics in Harmonic Maps vol. 50. American Mathematical Soc

  35. Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation 15(6):1373–1396

    Article  Google Scholar 

  36. Lutkepohl H (1997) Handbook of matrices. Computa Statistics Data analysis 2(25):243

    Google Scholar 

  37. Hartigan JA, Wong MA et al (1979) A k-means clustering algorithm. Applied statistics 28(1):100–108

    Article  Google Scholar 

  38. Ng A, Jordan M, Weiss Y (2001) On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems 14

  39. Elhamifar E, Vidal R (2013) Sparse subspace clustering: Algorithm, theory, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(11):2765–2781

    Article  Google Scholar 

  40. Ding S, Zhang N, Zhang J, Xu X, Shi Z (2017) Unsupervised extreme learning machine with representational features. Int J Machine Learn Cyber 8(2):587–595

    Article  Google Scholar 

  41. Papadimitriou CH, Steiglitz K (1998) Combinatorial Optimization: Algorithms and Complexity. Courier Corporation

  42. Yang Y, Shen H, Nie F, Ji R, Zhou X (2011) Nonnegative spectral clustering with discriminative regularization. Proceedings of the AAAI Conference on Artificial Intelligence 25:555–560

Download references

Acknowledgements

This work was supported by the Fundamental Research Grant Scheme (FRGS) under the Ministry of Higher Education (MOHE) with project number FRGS/1/2020/ICT02/MUSM/03/6.

Author information

Authors and Affiliations

  1. School of Information Technology, Monash University Malaysia, Bandar Sunway, 47500, Selangor, Malaysia

    Sharana Dharshikgan Suresh Dass, Ganesh Krishnasamy, Raveendran Paramesran & Raphaël C.-W. Phan

Authors
  1. Sharana Dharshikgan Suresh Dass
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ganesh Krishnasamy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Raveendran Paramesran
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raphaël C.-W. Phan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ganesh Krishnasamy.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dass, S.D.S., Krishnasamy, G., Paramesran, R. et al. Hessian unsupervised extreme learning machine. Int. J. Mach. Learn. & Cyber. 15, 2013–2022 (2024). https://doi.org/10.1007/s13042-023-02012-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-023-02012-3

Keywords

Search

Navigation