Skip to main content
SpringerLink
Log in

Convergence of gradient method for Elman networks

  • Published:
Applied Mathematics and Mechanics Aims and scope Submit manuscript

Abstract

The gradient method for training Elman networks with a finite training sample set is considered. Monotonicity of the error function in the iteration is shown. Weak and strong convergence results are proved, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. A numerical example is given to support the theoretical findings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Elman J L. Finding structure in time[J]. Cognitive Science, 1990, 14(2):179–211.

    Article  Google Scholar 

  2. Tsoi A C, Back A D. Locally recurrent globally feedforward networks: a critical review of architectures[J]. IEEE Transactions on Neural Networks, 1994, 5(2):229–239.

    Article  Google Scholar 

  3. Wang Deliang, Liu Xiaomei, Ahalt S C. On temporal generalization of simple recurrent networks[J]. Neural Networks, 1996, 9(7):1099–1118.

    Article  Google Scholar 

  4. Kremer S C. On the computational power of Elman-style recurrent networks[J]. IEEE Transactions on Neural Networks, 1995, 6(4):1000–1004.

    Article  Google Scholar 

  5. Pham D T, Liu X. Training of Elman networks and dynamic system modeling[J]. International Journal of Systems Science, 1996, 27(2):221–226.

    Article  MATH  Google Scholar 

  6. Cartling B. On the implicit acquisition of a context-free grammar by a simple recurrent neural network[J]. Neurocomputing, 2008, 71(7–9):1527–1537.

    Article  Google Scholar 

  7. Li Xiang, Chen Zengqiang, Yuan Zhuzhi, et al. Generating chaos by an Elman network[J]. IEEE Transactions on Circuits and Systems-I, 2001, 48(9):1126–1131.

    Article  MATH  MathSciNet  Google Scholar 

  8. Ekici S, Yildirim S, Poyraz M. A transmission line fault locator based on Elman recurrent networks[J]. Applied Soft Computing, 2008, DOI: 10.1016/j.asoc.2008.04.011.

  9. Neto L B, Coelho P H G, Soares de Mello J C C B, et al. Flow estimation using an Elman networks[C]. In: Wunsch D, et al (eds). Proceedings of 2004 IEEE International Joint Conference on Neural Networks, IEEE Press, Budapest, Hungary, 2004, 831–836.

    Google Scholar 

  10. Demuth H B, Beale M H, Hagan M T. Neural network toolbox user’s guide[M]. Natick, MA: The Mathworks Inc, 2007.

    Google Scholar 

  11. Jesús O D, Hagan M T. Backpropation algorithms for a broad class of dynamic networks[J]. IEEE Transactions on Neural Networks, 2007, 18(1):14–27.

    Article  Google Scholar 

  12. Williams R J, Zisper D. A learning algorithm for continually running fully recurrent neural networks[J]. Neural Computation, 1989, 1(2):270–280.

    Article  Google Scholar 

  13. Ku C C, Lee K Y. Diagonal recurrent neural networks for dynamic systems control[J]. IEEE Transaction on Neural Networks, 1995, 6(1):144–156.

    Article  Google Scholar 

  14. Xu Dongpo, Li Zhengxue, Wu Wei, et al. Convergence of gradient descent algorithm for diagonal recurrent neural networks[C]. In: Cui Guangzhao, et al (eds). International Conference on Bio-Inspired Computing: Theories and Applications, IEEE Press, Zhengzhou, China, 2007.

    Google Scholar 

  15. Kuan C M, Hornik K, White H. A convergence results for learning in recurrent neural networks[J]. Neural Computation, 1994, 6(3):420–440.

    Article  Google Scholar 

  16. Wu Wei, Feng Guorui, Li Zhengxue, et al. Convergence of an online gradient method for BP neural networks[J]. IEEE Transactions on Neural Networks, 2005, 16(3):533–540.

    Article  Google Scholar 

  17. Wu Wei, Shao Hongmei, Qu Di. Strong convergence for gradient methods for BP networks training[C]. In: Zhao Mingsheng and Shi Zhongzhi (eds). Proceedings of 2005 International Conference on Neural Networks and Brains, IEEE Press, Beijing, China, 2005, 332–334.

    Google Scholar 

  18. Gori M, Maggini M. Optimal convergence of on-line backpropagation[J]. IEEE Transaction on Neural Networks, 1996, 7(1):251–254.

    Article  Google Scholar 

  19. Ortega J, Rheinboldt W. Iterative solution of nonlinear equations in several variables[M]. New York: Academic Press, 1970.

    Google Scholar 

  20. Yuan Yaxiang, Sun Wenyu. Optimization theory and methods[M]. Beijing: Science Press, 2001, 149 (in Chinese).

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Applied Mathematics, Dalian University of Technology, Dalian, 116024, Liaoning Province, P. R. China

    Wei Wu  (吴微), Dong-po Xu  (徐东坡) & Zheng-xue Li  (李正学)

Authors
  1. Wei Wu  (吴微)
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dong-po Xu  (徐东坡)
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zheng-xue Li  (李正学)
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Wu  (吴微).

Additional information

Communicated by GUO Xing-ming

Project supported by the National Natural Science Foundation of China (No. 10471017)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wu, W., Xu, Dp. & Li, Zx. Convergence of gradient method for Elman networks. Appl. Math. Mech.-Engl. Ed. 29, 1231–1238 (2008). https://doi.org/10.1007/s10483-008-0912-z

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10483-008-0912-z

Key words

Chinese Library Classification

2000 Mathematics Subject Classification

Search

Navigation