Skip to main content
SpringerLink
Log in

Sparse and Outlier Robust Extreme Learning Machine Based on the Alternating Direction Method of Multipliers

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Extreme learning machine (ELM) has been extensively researched for its fast training speed and powerful learning abilities. Entering the era of big data, large-scale learning tasks, the universality of noisy data and data with distributed storage pose considerable challenges to ELM. The outlier robust ELM (OR-ELM) is an important variant of ELM that dramatically improves the robustness of the model by introducing the \(\ell _1\)-norm in the error term. Nevertheless, the solution of OR-ELM is fully dense, which requires a large amount of storage space and computational resources for massive learning tasks. In this paper, we extended OR-ELM to the sparse and outlier robust ELM (SOR-ELM) based on the elastic-net theory that can simultaneously improve the sparsity and stability of the model. We also proposed a distributed version of SOR-ELM (DSOR-ELM) for handling data with distributed storage and large-scale learning tasks. In addition, an effective iterative algorithm, the alternating direction method of multipliers (ADMM), was employed to train our proposed models. Even though extending ADMM to multi-block issues is not straightforward, its convergence can still be ensured for training SOR-ELM and DSOR-ELM. Finally, extensive numerical experiments demonstrate the superiority of SOR-ELM and DSOR-ELM in training data with outliers and distributed learning environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Jordan MI, Mitchell TM (2015) Machine learning: trends, perspectives, and prospects. Science 349(6245):255–260

    MathSciNet  MATH  Google Scholar 

  2. Biamonte J, Wittek P, Pancotti N, Rebentrost P, Wiebe N, Lloyd S (2017) Quantum machine learning. Nature 549(7671):195–202

    Google Scholar 

  3. Qiu J, Wu Q, Ding G, Xu Y, Feng S (2016) A survey of machine learning for big data processing. EURASIP J Adv Signal Process 2016(1):1–16

    Google Scholar 

  4. Yu Y, Si X, Hu C, Zhang J (2019) A review of recurrent neural networks: LSTM cells and network architectures. Neural Comput 31(7):1235–1270

    MathSciNet  MATH  Google Scholar 

  5. Rawat W, Wang Z (2017) Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput 29(9):2352–2449

    MathSciNet  MATH  Google Scholar 

  6. Ding S, Zhang N, Zhang X, Wu F (2017) Twin support vector machine: theory, algorithm and applications. Neural Comput Appl 28(11):3119–3130

    Google Scholar 

  7. Xing H, Xiao Z, Zhan D, Luo S, Dai P, Li K (2022) Selfmatch: robust semisupervised time-series classification with self-distillation. Int J Intell Syst 37(11):8583–8610

    Google Scholar 

  8. Xing H, Xiao Z, Qu R, Zhu Z, Zhao B (2022) An efficient federated distillation learning system for multi-task time series classification. IEEE Trans Instrum Meas 71:1–12

    Google Scholar 

  9. Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE international joint conference on neural networks (IEEE Cat. No. 04CH37541), vol 2, pp 985–990 . IEEE

  10. Huang G-B, Zhu Q-Y, Siew C-K (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1–3):489–501

    Google Scholar 

  11. Zhu S, Wang H, Lv H, Zhang H (2021) Augmented online sequential quaternion extreme learning machine. Neural Process Lett 53(2):1161–1186

    Google Scholar 

  12. Eshtay M, Faris H, Obeid N (2019) Metaheuristic-based extreme learning machines: a review of design formulations and applications. Int J Mach Learn Cybern 10(6):1543–1561

    Google Scholar 

  13. Perales-González C (2021) Global convergence of negative correlation extreme learning machine. Neural Process Lett 53(3):2067–2080

    Google Scholar 

  14. Albadr MAA, Tiun S (2020) Spoken language identification based on particle swarm optimisation-extreme learning machine approach. Circuits Syst Signal Process 39(9):4596–4622

    Google Scholar 

  15. Xu X, Deng J, Coutinho E, Wu C, Zhao L, Schuller BW (2018) Connecting subspace learning and extreme learning machine in speech emotion recognition. IEEE Trans Multimed 21(3):795–808

    Google Scholar 

  16. Ma J, Yang L, Wen Y, Sun Q (2020) Twin minimax probability extreme learning machine for pattern recognition. Knowl Based Syst 187:104806

    Google Scholar 

  17. Wang S-J, Chen H-L, Yan W-J, Chen Y-H, Fu X (2014) Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Process Lett 39(1):25–43

    Google Scholar 

  18. Chen BL, Shen YY, Zhu GC, Yu YT, Ji M (2022) An empirical mode decomposition fuzzy forecast model for COVID-19. Neural Process Lett 1–22

  19. Jiang F, Zhu Q, Tian T (2022) Breast cancer detection based on modified Harris Hawks optimization and extreme learning machine embedded with feature weighting. Neural Process Lett 1–24

  20. Martínez-Martínez JM, Escandell-Montero P, Soria-Olivas E, Martín-Guerrero JD, Magdalena-Benedito R, GóMez-Sanchis J (2011) Regularized extreme learning machine for regression problems. Neurocomputing 74(17):3716–3721

    Google Scholar 

  21. Li G, Niu P (2013) An enhanced extreme learning machine based on ridge regression for regression. Neural Comput Appl 22(3):803–810

    MathSciNet  Google Scholar 

  22. Miche Y, Van Heeswijk M, Bas P, Simula O, Lendasse A (2011) TROP-ELM: a double-regularized ELM using LARS and Tikhonov regularization. Neurocomputing 74(16):2413–2421

    Google Scholar 

  23. Fakhr MW, Youssef ENS, El-Mahallawy MS (2015) L1-regularized least squares sparse extreme learning machine for classification. In: 2015 International conference on information and communication technology research (ICTRC), pp 222–225. IEEE

  24. Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B (Methodol) 58(1):267–288

    MathSciNet  MATH  Google Scholar 

  25. Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–499

    MathSciNet  MATH  Google Scholar 

  26. Shi X, Kang Q, An J, Zhou M (2021) Novel l1 regularized extreme learning machine for soft-sensing of an industrial process. IEEE Trans Ind Inf 18(2):1009–1017

    Google Scholar 

  27. Zhang K, Luo M (2015) Outlier-robust extreme learning machine for regression problems. Neurocomputing 151:1519–1527

    Google Scholar 

  28. Wang Z, Sui L, Xin J, Qu L, Yao Y (2020) A survey of distributed and parallel extreme learning machine for big data. IEEE Access 8:201247–201258

    Google Scholar 

  29. Wang Y, Dou Y, Liu X, Lei Y (2016) PR-ELM: parallel regularized extreme learning machine based on cluster. Neurocomputing 173:1073–1081

    Google Scholar 

  30. Ming Y, Zhu E, Wang M, Ye Y, Liu X, Yin J (2018) DMP-ELMS: data and model parallel extreme learning machines for large-scale learning tasks. Neurocomputing 320:85–97

    Google Scholar 

  31. Bi X, Zhao X, Wang G, Zhang P, Wang C (2015) Distributed extreme learning machine with kernels based on mapreduce. Neurocomputing 149:456–463

    Google Scholar 

  32. Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113

    Google Scholar 

  33. Xin J, Wang Z, Qu L, Yu G, Kang Y (2016) A-ELM*: Adaptive distributed extreme learning machine with mapreduce. Neurocomputing 174:368–374

    Google Scholar 

  34. Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J R Stat Soc Ser B (Stat Methodol) 67(2):301–320

    MathSciNet  MATH  Google Scholar 

  35. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J, et al (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends® Mach Learn 3(1):1–122

  36. Zhang C, Li H, Chen C, Qian Y, Zhou X (2020) Enhanced group sparse regularized nonconvex regression for face recognition. IEEE Trans Pattern Anal Mach Intell

  37. Li H, Zhang C, Jia X, Gao Y, Chen C (2021) Adaptive label correlation based asymmetric discrete hashing for cross-modal retrieval. IEEE Trans Knowl Data Eng

  38. Li D, Tian Y (2018) Improved least squares support vector machine based on metric learning. Neural Comput Appl 30(7):2205–2215

    Google Scholar 

  39. Chen C, He B, Ye Y, Yuan X (2016) The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math Program 155(1):57–79

    MathSciNet  MATH  Google Scholar 

  40. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323:533–536

    MATH  Google Scholar 

  41. Marquardt DW (1963) An algorithm for least-squares estimation of nonlinear parameters. J Soc Ind Appl Math 11(2):431–441

    MathSciNet  MATH  Google Scholar 

  42. Zhang Y, Dai Y, Wu Q (2022) An accelerated optimization algorithm for the elastic-net extreme learning machine. Int J Mach Learn Cybern 13(12):3993–4011

    Google Scholar 

  43. da Silva BLS, Inaba FK, Salles EOT, Ciarelli PM (2020) Outlier robust extreme machine learning for multi-target regression. Expert Syst Appl 140:112877

    Google Scholar 

  44. Luo M, Zhang L, Liu J, Guo J, Zheng Q (2017) Distributed extreme learning machine with alternating direction method of multiplier. Neurocomputing 261:164–170

    Google Scholar 

  45. Donoho DL (1995) De-noising by soft-thresholding. IEEE Trans Inf Theory 41(3):613–627

    MathSciNet  MATH  Google Scholar 

  46. Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2:1–27

    Google Scholar 

  47. Dua D, Graff C (2017) UCI machine learning repository. http://archive.ics.uci.edu/ml

  48. Torgo L (2017) Regression data sets. https://www.dcc.fc.up.pt/~ltorgo/Regression/DataSets.html

  49. Yıldırım H, Revan Özkale M (2021) LL-ELM: a regularized extreme learning machine based on l_1-norm and Liu estimator. Neural Comput Appl 33(16):10469–10484

    Google Scholar 

  50. Lipu MSH, Hannan MA, Hussain A, Saad MH, Ayob A, Uddin MN (2019) Extreme learning machine model for state-of-charge estimation of lithium-ion battery using gravitational search algorithm. IEEE Trans Ind Appl 55(4):4225–4234

    Google Scholar 

  51. Guan L, Sun T, Qiao LB, Yang ZH, Li DS, Ge KS, Lu XC (2020) An efficient parallel and distributed solution to nonconvex penalized linear SVMs. Front Inf Technol Electron Eng 21(4):587–603

    Google Scholar 

Download references

Funding

This paper is supported by the National Natural Science Foundation of China (Grant No. 12271479).

Author information

Authors and Affiliations

  1. School of Mathematical Sciences, Zhejiang University, 38 Zheda Road, Hangzhou, 310027, Zhejiang, China

    Yuao Zhang, Yunwei Dai & Qingbiao Wu

Authors
  1. Yuao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunwei Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qingbiao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qingbiao Wu.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Dai, Y. & Wu, Q. Sparse and Outlier Robust Extreme Learning Machine Based on the Alternating Direction Method of Multipliers. Neural Process Lett 55, 9787–9809 (2023). https://doi.org/10.1007/s11063-023-11227-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11227-y

Keywords

Search

Navigation