Skip to main content
SpringerLink
Log in

Distributed regularized stochastic configuration networks via the elastic net

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Stochastic configuration network (SCN) has great potential in developing fast learning model with sound generalization capability and can be easily extended to the distributed computing framework. This paper aims to develop a distributed regularized stochastic configuration network to solve the limitations of traditional centralized learning on the scalability and efficiency in computing and storage resources for massive datasets. The local models are constructed using a classical stochastic configuration network, and the global unified model is built by the alternating direction method of multipliers (ADMM). Elastic net regularization term combining the LASSO and ridge methods is added into loss function of the ADMM optimization to prevent the model from overfitting when the data has high-dimensional collinearity. Each layer of the local regularized SCN model of a node in the topology network is constructed incrementally; its input weights and biases are broadcast to all other nodes under the inequality constraints. Output weights and the Lagrange multipliers of each node are calculated alternately through the decomposition–coordination procedure of the ADMM optimization algorithm until it finally converges to a unified model. A comprehensive study on five benchmark datasets and the ball mill experimental data has been carried out to evaluate the proposed method. The experiment results show that the proposed distributed regularized stochastic configuration network has relative advantages in terms of accuracy and stability compared with the distributed random vector functional link network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://github.com/ispamm/Lynx-Toolbox.

  2. http://people.cs.uchicago.edu/~vikass/manifoldregularization.html.

  3. http://www.causality.inf.ethz.ch/al_data/SYLVA.html.

References

  1. Gupta D, Rani R (2019) A study of big data evolution and research challenges. J Inf Sci 45(3):322–340

    Article  Google Scholar 

  2. Zhou L, Pan S, Wang J, Vasilakos AV (2017) Machine learning on big data: opportunities and challenges. Neurocomputing 237:350–361

    Article  Google Scholar 

  3. Wu X, Zhu X, Wu G-Q, Ding W (2014) Data mining with big data. IEEE Trans Knowl Data Eng 26(1):97–107

    Article  Google Scholar 

  4. Peteiro-Barral D, Guijarro-Berdiñas B (2013) A survey of methods for distributed machine learning. Prog Artif Intell 2(1):1–11

    Article  Google Scholar 

  5. Galakatos A, Crotty A, Kraska T (2018) Distributed machine learning. Springer New York, New York, pp 1196–1201

    Google Scholar 

  6. Fang Y, Jia Q, Guo L, Wang G (2019) Efficient privacy-preserving machine learning in hierarchical distributed system. IEEE Trans Netw Sci Eng 6(4):599–612

    Article  MathSciNet  Google Scholar 

  7. Amiri MM, Gündüz D (2019) Computation scheduling for distributed machine learning with straggling workers. IEEE Trans Signal Process 67(24):6270–6284

    Article  MathSciNet  Google Scholar 

  8. Taylor G, Burmeister R, Xu Z, Singh B, Patel A, Goldstein T (2016) Training neural networks without gradients: a scalable admm approach. In: International conference on machine learning, pp 2722–2731

  9. Scardapane S, Wang D, Panella M (2016) A decentralized training algorithm for echo state networks in distributed big data applications. Neural Netw 78:65–74

    Article  Google Scholar 

  10. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J et al (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Mach Learn 3(1):1–122

    Article  Google Scholar 

  11. Glowinski R, Marrocco A (1975) On the solution of a class of non linear dirichlet problems by a penalty-duality method and finite elements of order one. In: Optimization techniques IFIP technical conference. Springer, pp 327–333

  12. Gabay D, Mercier B (1976) A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput Math Appl 2(1):17–40

    Article  Google Scholar 

  13. Mahoney MW et al (2011) Randomized algorithms for matrices and data. Found Trends Mach Learn 3(2):123–224

    MATH  Google Scholar 

  14. Scardapane S, Wang D (2017) Randomness in neural networks: an overview. Wiley Interdiscip Rev Data Min Knowl Discov 7(2):e1200

    Article  Google Scholar 

  15. Igelnik B, Pao Y-H (1995) Stochastic choice of basis functions in adaptive function approximation and the functional-link net. IEEE Trans Neural Netw 6(6):1320–1329

    Article  Google Scholar 

  16. Scardapane S, Wang D, Panella M, Uncini A (2015) Distributed learning for random vector functional-link networks. Inf Sci 301:271–284

    Article  MathSciNet  Google Scholar 

  17. Li M, Wang D (2017) Insights into randomized algorithms for neural networks: practical issues and common pitfalls. Inf Sci 382:170–178

    Article  Google Scholar 

  18. Wang D, Li M (2017) Stochastic configuration networks: fundamentals and algorithms. IEEE Trans Cybern 47(10):3466–3479

    Article  Google Scholar 

  19. Huang C, Huang Q, Wang D (2019) Stochastic configuration networks based adaptive storage replica management for power big data processing. IEEE Trans Ind Inform 16(1):373–383

    Article  Google Scholar 

  20. Wang D, Li M (2017) Robust stochastic configuration networks with kernel density estimation for uncertain data regression. Inf Sci 412:210–222

    Article  MathSciNet  Google Scholar 

  21. Hoerl AE, Kennard RW (2000) Ridge regression: biased estimation for nonorthogonal problems. Technometrics 42(1):80–86

    Article  Google Scholar 

  22. Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J R Stat Soc Ser B (Stat Methodol) 67(2):301–320

    Article  MathSciNet  Google Scholar 

  23. Gilbert EN (1959) Random graphs. Ann Math Stat 30(4):1141–1144

    Article  Google Scholar 

  24. Mierswa I, Morik K (2005) Automatic feature extraction for classifying audio data. Mach Learn 58(2–3):127–149

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Key R&D Program of China under Grant 2018YFB1700200.

Author information

Authors and Affiliations

  1. College of Information Engineering, Shenyang University of Chemical Technology, Shenyang, 110142, Liaoning, China

    Lijie Zhao, Shida Zou, Mingzhong Huang & Guogang Wang

Authors
  1. Lijie Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shida Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingzhong Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guogang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingzhong Huang.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest in the research.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, L., Zou, S., Huang, M. et al. Distributed regularized stochastic configuration networks via the elastic net. Neural Comput & Applic 33, 3281–3297 (2021). https://doi.org/10.1007/s00521-020-05178-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05178-x

Keywords

Search

Navigation