Skip to main content
SpringerLink
Log in

Deep echo state network with reservoirs of multiple activation functions for time-series prediction

  • Published:
Sādhanā Aims and scope Submit manuscript

Abstract

In this paper, an improved deep echo state network is proposed, named as multiple activation functions deep echo state network (MAF-DESN), where states are activated by multiple activation functions. A sufficient condition for MAF-DESN is given to guarantee that MAF-DESN possesses the echo state property. Finally, the MAF-DESN is applied to chaotic time-series predictions and compared to other ESN deformation models and popular LSTM. Simulation results show that under same network size condition, MAF-DESN possesses stronger explanatory power in chaotic far-infrared laser predictions (R-square=0.9537, others≤0.6487), and better fitting ability in daily foreign exchange rates (MAE=0.0040, others≥0.0047) and chaotic far-infrared laser (MAE=3.4042, others≥4.9021). In high-dimension-input task, MAF-DESN improved the performance when the results were compared (R-square=0.4274, others≤0.3975 and MAE=5.2221, others≥7.6876), while the train time of MAF-DESN did not increase when compared to DESN.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10

Similar content being viewed by others

References

  1. Schrauwen B et al 2008 Improving reservoirs using intrinsic plasticity. Neurocomputing 71: 1159–1171

    Article  Google Scholar 

  2. Guo Y et al 2017 Robust echo state networks based on correntropy induced loss function. Neurocomputing 267: 295–303

    Article  Google Scholar 

  3. Chen L, Qu H and Zhao J 2016 Generalized correntropy induced loss function for deep learning. In: Proceedings of the International Joint Conference on Neural Networks, IEEE, pp. 1428–1433

  4. Han M and Xu M 2018 Laplacian echo state network for multivariate time series prediction. IEEE Trans. Neural Netw. Learn. Syst. 29(1): 238–244

    Article  MathSciNet  Google Scholar 

  5. Basterrech S and Rubino G 2017 Echo state queueing networks: a combination of reservoir computing and random neural networks. Probab. Eng. Inf. Sci. 31(4): 1–20

    Article  MathSciNet  Google Scholar 

  6. Dorado-Moreno M et al 2016 Multiclass prediction of wind power ramp events combining reservoir computing and support vector machines. Adv. Artif. Intell. 1: 300–309

    Article  Google Scholar 

  7. Sun X et al 2017 Deep belief echo-state network and its application to time series prediction. Knowl. Based Syst. 130: 17–29

    Article  Google Scholar 

  8. Evermann J, Rehse J R and Fettke P 2017 Predicting process behaviour using deep learning. Decis. Support Syst. 100: 129–140

    Article  Google Scholar 

  9. Wang H and Yan X 2015 Optimizing the echo state network with a binary particle swarm optimization algorithm. Knowl. Based Syst. 86: 182–193

    Article  Google Scholar 

  10. Chan P K et al 2017 Sensitivity based robust learning for stacked autoencoder against evasion attack. Neurocomputing 267: 572–580

    Article  Google Scholar 

  11. Yang C et al 2018 Dynamical regularized echo state network for time series prediction. Neural Comput. Appl. 1–14, https://doi.org/10.1007/s00521-018-3488-z

  12. Qiao J, Wang L and Yang C 2018 Adaptive lasso echo state network based on modified Bayesian information criterion for nonlinear system modeling. Neural Comput. Appl. 1–15, https://doi.org/10.1007/s00521-018-3420-6

  13. Huang B et al 2018 Recursive Bayesian echo state network with an adaptive inflation factor for temperature prediction. Neural Comput. Appl. 29(12): 1–9

    Article  Google Scholar 

  14. Yang C et al 2018 Design of polynomial echo state networks for time series prediction. Neurocomputing 290: 148–160

    Article  Google Scholar 

  15. Malik Z K, Hussain A and Wu Q J 2017 Multilayered echo state machine: a novel architecture and algorithm. IEEE Trans. Cybern. 47(4): 1–14

    Article  Google Scholar 

  16. Gallicchio C, Micheli A and Pedrelli L 2017 Deep reservoir computing: a critical experimental analysis. Neurocomputing 268: 87–99

    Article  Google Scholar 

  17. Escalona-Morán M A et al 2015 Electrocardiogram classification using reservoir computing with logistic regression. IEEE J. Biomed. Health Inf. 19(3): 892–898

    Article  Google Scholar 

  18. Xiang L et al 2017 Deep auto-context convolutional neural networks for standard-dose pet image estimation from low-dose pet/mri. Neurocomputing 267: 406–416

    Article  Google Scholar 

  19. Dhungel N, Carneiro G and Bradley A P 2017 A deep learning approach for the analysis of masses in mammograms with minimal user intervention. Med. Image Anal. 37: 114–128

    Article  Google Scholar 

  20. Lu Y et al 2017 Identification of rice diseases using deep convolutional neural networks. Neurocomputing 267: 378–384

    Article  Google Scholar 

  21. Affonso C et al 2017 Deep learning for biological image classification. Expert Syst. Appl. 85: 114–122

    Article  Google Scholar 

  22. Liang L, Liu M and Sun W 2017 A deep learning approach to estimate chemically-treated collagenous tissue nonlinear anisotropic stress–strain responses from microscopy images. Acta Biomater. 63: 227–235

    Article  Google Scholar 

  23. Zheng Y J et al 2017 Airline passenger profiling based on fuzzy deep machine learning. IEEE Trans. Neural Netw. Learn. Syst. 28: 2911–2923

    Article  MathSciNet  Google Scholar 

  24. Gopalakrishnan K et al 2017 Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection. Constr. Build. Mater. 157: 322–330

    Article  Google Scholar 

  25. Deng W et al 2017 Deep correlation feature learning for face verification in the wild. IEEE Signal Process. Lett. 24(12): 1877–1881

    Article  Google Scholar 

  26. Arcos-García Á et al 2017 Exploiting synergies of mobile mapping sensors and deep learning for traffic sign recognition systems. Expert Syst. Appl. 89: 286–295

    Article  Google Scholar 

  27. Chen L, Qu H and Zhao J 2018 Generalized Correntropy based deep learning in presence of non-Gaussian noises. Neurocomputing 278: 41–50

    Article  Google Scholar 

  28. Qi T et al 2017 Image-based action recognition using hint-enhanced deep neural networks. Neurocomputing 267: 475–488

    Article  Google Scholar 

  29. Lun S X et al 2015 A novel model of leaky integrator echo state network for time-series prediction. Neurocomputing 159(1): 58–66

    Article  Google Scholar 

  30. Jaeger H 2010 The “echo state” approach to analysing and training recurrent neural networks – with an Erratum note. GMD Report 148, pp. 1–47

    Google Scholar 

Download references

Acknowledgements

This work is supported by Sichuan Science and Technology Program (2019YFSY0016). The authors declare that there is no conflict of interest regarding the publication of this paper.

Author information

Authors and Affiliations

  1. School of Electronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China

    YONGBO LIAO & HONGMEI LI

  2. State Key Laboratory of Electronic Thin Films and Integrated Devices, University of Electronic Science and Technology of China, Chengdu, China

    YONGBO LIAO

Authors
  1. YONGBO LIAO
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. HONGMEI LI
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to YONGBO LIAO.

Appendices

Appendix A

Before proving that \( \left\| {\varvec{x}\left( {n + 1} \right) - \varvec{x}^{{\mathbf{\prime }}} \left( {n + 1} \right)} \right\| \le \left\| {\varvec{x}\left( n \right) - \varvec{x}^{{\mathbf{\prime }}} \left( n \right)} \right\| \), we describe the relationship between \( \left\| {\varvec{x}\left( {n + 1} \right) - \varvec{x}^{{\mathbf{\prime }}} \left( {n + 1} \right)} \right\| \) and \( \left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\| \).

“(a)”: According to aforementioned gathering of reservoir states \( \varvec{x}\left( n \right) \) and reservoir states of \( l \)th layer \( \varvec{x}^{\left( l \right)} \left( n \right) \), we unfold \( \varvec{x}\left( n \right) \) as the following vector:

$$ \begin{aligned} & \varvec{x}\left( n \right) = \\ & \left[ {\begin{array}{*{20}c} {\overbrace {{\begin{array}{*{20}c} {x_{1}^{\left( 1 \right)} \left( n \right)} & \cdots & {x_{{N_{1} }}^{\left( 1 \right)} \left( n \right)} \\ \end{array} }}^{{\left( {\varvec{x}^{\left( 1 \right)} \left( n \right)} \right)^{T} }}} & \cdots & {\overbrace {{\begin{array}{*{20}c} {x_{1}^{\left( L \right)} \left( n \right)} & \cdots & {x_{{N_{L} }}^{\left( L \right)} \left( n \right)} \\ \end{array} }}^{{\left( {\varvec{x}^{\left( L \right)} \left( n \right)} \right)^{T} }}} \\ \end{array} } \right] \\ \end{aligned} $$
(A.1)

where superscript denotes reservoir location, the maximum number of layers is L, subscript denotes neuron location of current reservoir and maximum number of reservoir neurons is Nl (\( l = 1, \cdots L \)). Then, substituting Eq. (A.1) into \( \left\| {\varvec{x}\left( {n + 1} \right) - \varvec{x}^{{\mathbf{\prime }}} \left( {n + 1} \right)} \right\| \), we obtain following equations:

$$ \begin{aligned} & \left\| {\varvec{x}\left( {n + 1} \right) - \varvec{x}^{{\mathbf{\prime }}} \left( {n + 1} \right)} \right\| \\ & \quad = \left\| {\begin{array}{*{20}c} {x_{1}^{\left( 1 \right)} \left( {n + 1} \right) - x_{1}^{\prime \left( 1 \right)} \left( {n + 1} \right)} \\ \vdots \\ {x_{{N_{1} }}^{\left( 1 \right)} \left( {n + 1} \right) - x_{{N_{1} }}^{\prime \left( 1 \right)} \left( {n + 1} \right)} \\ \vdots \\ {x_{1}^{\left( L \right)} \left( {n + 1} \right) - x_{1}^{\prime \left( L \right)} \left( {n + 1} \right)} \\ \vdots \\ {x_{{N_{L} }}^{\left( L \right)} \left( {n + 1} \right) - x_{{N_{L} }}^{\prime \left( L \right)} \left( {n + 1} \right)} \\ \end{array} } \right\| \\ & \quad = \sqrt {\sum\nolimits_{l = 1}^{L} {\sum\nolimits_{j = 1}^{{N_{l} }} {\left( {x_{j}^{\left( l \right)} \left( {n + 1} \right) - x_{j}^{\prime \left( l \right)} \left( {n + 1} \right)} \right)^{2} } } } , \\ \end{aligned} $$
(A.2)
$$ \begin{aligned} & \left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{\varvec{'}\left( l \right)}} \left( {n + 1} \right)} \right\| \\ & \quad \quad = \left\| {\begin{array}{*{20}c} {x_{1}^{\left( l \right)} \left( {n + 1} \right) - x_{1}^{'\left( l \right)} \left( {n + 1} \right)} \\ \vdots \\ {x_{{N_{l} }}^{\left( l \right)} \left( {n + 1} \right) - x_{{N_{l} }}^{'\left( l \right)} \left( {n + 1} \right)} \\ \end{array} } \right\| \\ & \quad \quad \quad = \sqrt {\sum\nolimits_{j = 1}^{{N_{l} }} {\left( {x_{j}^{\left( l \right)} \left( {n + 1} \right) - x_{j}^{'\left( l \right)} \left( {n + 1} \right)} \right)^{2} } } . \\ \end{aligned} $$
(A.3)

According to Eqs. (A.2) and (A.3), we obtain the relationship between \( \left\| {\varvec{x}\left( {n + 1} \right) - \varvec{x}^{{\mathbf{\prime }}} \left( {n + 1} \right)} \right\| \) and \( \left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\| \) as follows:

$$ \begin{aligned} & \left\| {\varvec{x}\left( {n + 1} \right) - \varvec{x}^{{\mathbf{\prime }}} \left( {n + 1} \right)} \right\|^{2} \\ & \quad \quad \quad \quad = \sum\nolimits_{l = 1}^{L} {\left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} } . \\ \end{aligned} $$
(A.4)

“(b)”: Now, the proof of Inequation (9) is described here. According to Eq. (A.4), proof of \( \left\| {\varvec{x}\left( {n + 1} \right) - \varvec{x}^{{\mathbf{\prime }}} \left( {n + 1} \right)} \right\| \le \left\| {\varvec{x}\left( n \right) - \varvec{x}^{{\mathbf{\prime }}} \left( n \right)} \right\| \) is equivalent to that of

$$ \begin{aligned} & \sum\nolimits_{l = 1}^{L} {\left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} } \\ & \quad \quad \quad \quad \quad \quad \le \sum\nolimits_{l = 1}^{L} {\left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|^{2} } . \\ \end{aligned} $$
(A.5)

Hence, we prove the latter inequation as follows:

$$ \begin{aligned} & \sum\nolimits_{l = 1}^{L} {\left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} } \\ & \quad = \sum\nolimits_{l = 1}^{L} {\left\| {\left( {1 - a^{\left( l \right)} } \right)\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right.} \\ & \quad \quad \left. { + a^{\left( l \right)} \left( {f^{\left( l \right)} \left( {\varvec{x}_{{\varvec{in}}}^{\left( l \right)} \left( {n + 1} \right)} \right) - f^{\left( l \right)} \left( {\varvec{x}_{{\varvec{in}}}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right)} \right\|^{2} \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left\| {\left( {1 - a^{\left( l \right)} } \right)\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right.} \\ & \quad \quad \left. { + a^{\left( l \right)} \left( {\varvec{x}_{{\varvec{in}}}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}_{{\varvec{in}}}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right\|^{2} \\ & \quad = \sum\nolimits_{l = 1}^{L} {\left\| {\left( {\left( {1 - a^{\left( l \right)} } \right)\varvec{E}^{\left( l \right)} + a^{\left( l \right)} W^{\left( l \right)} } \right)\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\prime }\left( l \right)}} \left( n \right)} \right)} \right.} \\ & \quad \quad \left. { + a^{\left( l \right)} \varvec{W}_{{\varvec{in}}}^{\left( l \right)} \left( {\varvec{i}^{\left( l \right)} \left( {n + 1} \right) - \varvec{i}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right\|^{2} \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left( {\left\| {\left( {\left( {1 - a^{\left( l \right)} } \right)\varvec{E}^{\left( l \right)} + a^{\left( l \right)} \varvec{W}^{\left( l \right)} } \right)\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right\|} \right.} \\ & \quad \quad \left. { + \left\| {a^{\left( l \right)} \varvec{W}_{{\varvec{in}}}^{\left( l \right)} \left( {\varvec{i}^{\left( l \right)} \left( {n + 1} \right) - \varvec{i}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right\|} \right)^{2} \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left\| {\left( {\left( {1 - a^{\left( l \right)} } \right)\varvec{E}^{\left( l \right)} + a^{\left( l \right)} \varvec{W}^{\left( l \right)} } \right)\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right\|^{2} } \\ & \quad \quad + \sum\nolimits_{l = 1}^{L} {\left\| {a^{\left( l \right)} \varvec{W}_{{\varvec{in}}}^{\left( l \right)} \left( {\varvec{i}^{\left( l \right)} \left( {n + 1} \right) - \varvec{i}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right\|^{2} } . \\ \end{aligned} $$
(A.6)

According to Eq. (2), we know that

$$ \left\{ {\begin{array}{*{20}l} {\varvec{i}^{\left( l \right)} \left( {n + 1} \right) - \varvec{i}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right) = 0,l = 1} \hfill \\ {\varvec{i}^{\left( l \right)} \left( {n + 1} \right) - \varvec{i}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right) = \varvec{x}^{{\left( {l - 1} \right)}} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( {l - 1} \right)}} \left( {n + 1} \right),l > 1} \hfill \\ \end{array} } \right. $$
(A.7)

Then

$$ \begin{aligned} & \left\| {\varvec{x}\left( {n + 1} \right) - \varvec{x}^{{\mathbf{\prime }}} \left( {n + 1} \right)} \right\|^{2} = \sum\nolimits_{l = 1}^{L} {\left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} } \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left\| {\left( {\left( {1 - a^{\left( l \right)} } \right)\varvec{E}^{\left( l \right)} + a^{\left( l \right)} \varvec{W}^{\left( l \right)} } \right)\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right\|^{2} } \\ & \quad \quad + \sum\nolimits_{l = 2}^{L} {\left\| {a^{\left( l \right)} \varvec{W}_{{\varvec{in}}}^{\left( l \right)} \left( {\varvec{x}^{{\left( {l - 1} \right)}} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( {l - 1} \right)}} \left( {n + 1} \right)} \right)} \right\|^{2} } \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left\| {\left( {\left( {1 - a^{\left( l \right)} } \right)\varvec{E}^{\left( l \right)} + a^{\left( l \right)} \varvec{W}^{\left( l \right)} } \right)\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right\|^{2} } \\ & \quad \quad + \sum\nolimits_{l = 1}^{L - 1} {\left\| {a^{{\left( {l + 1} \right)}} \varvec{W}_{{\varvec{in}}}^{{\left( {l + 1} \right)}} \left( {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right\|^{2} } \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left( {1 - a^{\left( l \right)} + a^{\left( l \right)} \varLambda^{\left( l \right)} } \right)^{2} \left\| {\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right\|^{2} } \\ & \quad \quad + \sum\nolimits_{l = 1}^{L - 1} {\left( {a^{{\left( {l + 1} \right)}} \varLambda_{\text{in}}^{{\left( {l + 1} \right)}} } \right)^{2} \left\| {\left( {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right\|^{2} } \\ & \Rightarrow \sum\nolimits_{l = 1}^{L} {\left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} } \\ & & \quad \quad \quad - \sum\nolimits_{l = 1}^{L - 1} {\left( {a^{{\left( {l + 1} \right)}} \varLambda_{\text{in}}^{{\left( {l + 1} \right)}} } \right)^{2} \left\| {\left( {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right\|^{2} } \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left( {1 - a^{\left( l \right)} + a^{\left( l \right)} \varLambda^{\left( l \right)} } \right)^{2} \left\| {\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right\|^{2} } \\ & \Rightarrow \sum\nolimits_{l = 1}^{L - 1} {\left( {1 - \left( {a^{{\left( {l + 1} \right)}} \varLambda_{\text{in}}^{{\left( {l + 1} \right)}} } \right)^{2} } \right)\left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} } \\ & \quad \quad \quad + \left\| {\varvec{x}^{\left( L \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( L \right)}} \left( {n + 1} \right)} \right\|^{2} \\ & \le \sum\nolimits_{l = 1}^{L} {\left( {1 - a^{\left( l \right)} + a^{\left( l \right)} \varLambda^{\left( l \right)} } \right)^{2} \left\| {\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right\|^{2} } . \\ \end{aligned} $$
(A.8)

Unfolding of Inequation (A.8) is as follows: when \( l{ = }1,2, \cdots L - 1 \)

$$ \begin{aligned} & \left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} \\ & \quad \quad \le \frac{{\left( {1 - a^{\left( l \right)} + a^{\left( l \right)} \varLambda^{\left( l \right)} } \right)^{2} }}{{1 - \left( {a^{{\left( {l + 1} \right)}} \varLambda_{\text{in}}^{{\left( {l + 1} \right)}} } \right)^{2} }}\left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|^{2} . \\ \end{aligned} $$
(A.9)

When \( l{ = }L \)

$$ \begin{aligned} & \left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} \\ & \quad \quad \le \left( {1 - a^{\left( l \right)} + a^{\left( l \right)} \varLambda^{\left( l \right)} } \right)^{2} \left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|^{2} . \\ \end{aligned} $$
(A.10)

According to Inequations (A.9) and (A.10), when following Inequations (A.11) and (A.12)

$$ \begin{aligned} & \frac{{\left( {1 - a^{\left( l \right)} + a^{\left( l \right)} \varLambda^{\left( l \right)} } \right)^{2} }}{{1 - \left( {a^{{\left( {l + 1} \right)}} \varLambda_{\text{in}}^{{\left( {l + 1} \right)}} } \right)^{2} }}\left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|^{2} \\ & \quad \quad \quad \le \left\| {\varvec{x}^{\left( L \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( L \right)}} \left( n \right)} \right\|^{2} ,l = 1,2, \cdots L - 1 \\ \end{aligned} $$
(A.11)
$$ \begin{aligned} & \left( {1 - a^{\left( l \right)} + a^{\left( l \right)} \varLambda^{\left( l \right)} } \right)^{2} \left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|^{2} \\ & \quad \quad \le \left\| {\varvec{x}^{\left( L \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( L \right)}} \left( {n + 1} \right)} \right\|^{2} ,l = L \\ \end{aligned} $$
(A.12)

are satisfied, then

$$ \left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} \le \left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|^{2} $$
(A.13)

are satisfied, which means that

$$ \sum\nolimits_{l = 1}^{L} {\left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} } \le \sum\nolimits_{l = 1}^{L} {\left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|^{2} } $$
(A.14)

are satisfied, which also means that \( \left\| {\varvec{x}\left( {n + 1} \right) - \varvec{x}^{{\mathbf{\prime }}} \left( {n + 1} \right)} \right\|^{2} \le \left\| {\varvec{x}\left( n \right) - \varvec{x}^{{\mathbf{\prime }}} \left( n \right)} \right\|^{2} \) are satisfied. We transform Inequations (A.11) and (A.12) into the following form:

$$ \left\{ {\begin{array}{*{20}l} {\frac{{\left( {1 - a^{\left( l \right)} { + }a^{\left( l \right)} \varLambda^{\left( l \right)} } \right)^{2} }}{{1 - \left( {a^{{\left( {l + 1} \right)}} \varLambda_{\text{in}}^{{\left( {l + 1} \right)}} } \right)^{2} }} \le 1,l{ = }1,2, \cdots L - 1} \hfill \\ {\left( {1 - a^{\left( l \right)} { + }a^{\left( l \right)} \varLambda^{\left( l \right)} } \right)^{2} \le 1,l{ = }L} \hfill \\ \end{array} } \right. $$
(A.15)

Then, the proof process is completed.

Appendix B

According to Eq. (A.4), proof of Inequation (13) is equivalent to that of (A.14). Hence, we prove the latter Inequation (A.14) as follows:

$$ \begin{aligned} & \sum\nolimits_{l = 1}^{L} {\left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} } \\ & \quad = \sum\nolimits_{l = 1}^{L} {\left\| {\left( {1 - a^{\left( l \right)} } \right)\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right.} \\ & \quad \quad \left. { + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left( {\lambda_{i}^{\left( l \right)} \left( {f_{i}^{\left( l \right)} \left( {\varvec{x}_{{\varvec{in}}}^{\left( l \right)} \left( {n + 1} \right)} \right) - f_{i}^{\left( l \right)} \left( {\varvec{x}_{{\varvec{in}}}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right)} \right)} } \right\|^{2} \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left\| {\left( {1 - a^{\left( l \right)} } \right)\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right.} \\ & \quad \quad \left. { + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left( {\lambda_{i}^{\left( l \right)} \left( {\varvec{x}_{{\varvec{in}}}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}_{{\varvec{in}}}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right)} } \right\|^{2} \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left\| {\left( {\left( {1 - a^{\left( l \right)} } \right)\varvec{E}^{\left( l \right)} + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\lambda_{i}^{\left( l \right)} \varvec{W}^{\left( l \right)} } } \right)\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right.} \\ & \quad \quad \left. { + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left( {\lambda_{i}^{\left( l \right)} \varvec{W}_{{\varvec{in}}}^{\left( l \right)} \left( {\varvec{i}^{\left( l \right)} \left( {n + 1} \right) - \varvec{i}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right)} } \right\|^{2} \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left\| {\left( {\left( {1 - a^{\left( l \right)} } \right)\varvec{E}^{\left( l \right)} + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\lambda_{i}^{\left( l \right)} \varvec{W}^{\left( l \right)} } } \right)\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right\|}^{2} \\ & \quad \quad + \sum\nolimits_{l = 1}^{L} {\left\| {\sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left( {\lambda_{i}^{\left( l \right)} \varvec{W}_{{\varvec{in}}}^{\left( l \right)} \left( {\varvec{i}^{\left( l \right)} \left( {n + 1} \right) - \varvec{i}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right)} } \right\|}^{2} \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left( {1 - a^{\left( l \right)} + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } } \right)^{2} \left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|}^{2} \\ & \quad \quad + \sum\nolimits_{l = 1}^{L} {\left( {\sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda_{\text{in}}^{\left( l \right)} } } \right)^{2} \left\| {\varvec{i}^{\left( l \right)} \left( {n + 1} \right) - \varvec{i}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|}^{2} . \\ \end{aligned} $$
(B.1)

According to Eq. (A.7)

$$ \begin{aligned} & \left\| {\varvec{x}\left( {n + 1} \right) - \varvec{x}^{{\mathbf{\prime }}} \left( {n + 1} \right)} \right\|^{2} \\ & \quad = \sum\nolimits_{l = 1}^{L} {\left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} } \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left( {1 - a^{\left( l \right)} + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } } \right)^{2} \left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|}^{2} \\ & \quad \quad + \sum\nolimits_{l = 2}^{L} {\left( {\sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda_{\text{in}}^{\left( l \right)} } } \right)^{2} \left\| {\varvec{x}^{{\left( {l - 1} \right)}} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( {l - 1} \right)}} \left( {n + 1} \right)} \right\|}^{2} \\ & \quad \le \sum\nolimits_{l = 1}^{L} {\left( {1 - a^{\left( l \right)} + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } } \right)^{2} \left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|}^{2} \\ & \quad \quad + \sum\nolimits_{l = 1}^{L - 1} {\left( {\sum\nolimits_{i = 1}^{{N_{\lambda }^{l + 1} }} {\left| {\lambda_{i}^{{\left( {l + 1} \right)}} } \right|\varLambda_{\text{in}}^{{\left( {l + 1} \right)}} } } \right)^{2} \left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|}^{2} \\ \end{aligned} $$
(B.2)

Transform Inequation (B.2) as follows:

$$ \begin{aligned} & \sum\nolimits_{l = 1}^{L} {\left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} } \\ & \quad \quad - \sum\nolimits_{l = 1}^{L - 1} {\left( {\sum\nolimits_{i = 1}^{{N_{\lambda }^{l + 1} }} {\left| {\lambda_{i}^{{\left( {l + 1} \right)}} } \right|\varLambda_{\text{in}}^{{\left( {l + 1} \right)}} } } \right)^{2} \left\| {\left( {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right)} \right\|^{2} } \\ & \quad \quad \le \sum\nolimits_{l = 1}^{L} {\left( {1 - a^{\left( l \right)} + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } } \right)^{2} \left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|^{2} } \\ \end{aligned} $$
(B.3)

Transform Inequation (B.3) as follows:

$$ \begin{aligned} & \sum\nolimits_{l = 1}^{L - 1} {\left( {1 - \left( {\sum\nolimits_{i = 1}^{{N_{\lambda }^{l + 1} }} {\left| {\lambda_{i}^{{\left( {l + 1} \right)}} } \right|\varLambda_{\text{in}}^{{\left( {l + 1} \right)}} } } \right)^{2} } \right)\left\| {\varvec{x}^{\left( l \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( {n + 1} \right)} \right\|^{2} } \\ & \quad \quad + \left\| {\varvec{x}^{\left( L \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( L \right)}} \left( {n + 1} \right)} \right\|^{2} \\ & \quad \quad \le \sum\nolimits_{l = 1}^{L} {\left( {1 - a^{\left( l \right)} + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } } \right)^{2} \left\| {\left( {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right)} \right\|^{2} } \\ \end{aligned} $$
(B.4)

Similar to Appendix A, when the following inequations

$$ \begin{aligned} & \frac{{\left( {1 - a^{\left( l \right)} + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } } \right)^{2} }}{{1 - \left( {\sum\nolimits_{i = 1}^{{N_{\lambda }^{l + 1} }} {\left| {\lambda_{i}^{{\left( {l + 1} \right)}} } \right|\varLambda_{\text{in}}^{{\left( {l + 1} \right)}} } } \right)^{2} }}\left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|^{2} \\ & \quad \quad \le \left\| {\varvec{x}^{\left( L \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( L \right)}} \left( n \right)} \right\|^{2} ,l = 1,2, \cdots L - 1 \\ \end{aligned} $$
(B.5)
$$ \begin{aligned} & \left( {1 - a^{\left( l \right)} + \sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } } \right)^{2} \left\| {\varvec{x}^{\left( l \right)} \left( n \right) - \varvec{x}^{{{\mathbf{\prime }}\left( l \right)}} \left( n \right)} \right\|^{2} \\ & \quad \quad \quad \quad \quad \quad \quad \le \left\| {\varvec{x}^{\left( L \right)} \left( {n + 1} \right) - \varvec{x}^{{{\mathbf{\prime }}\left( L \right)}} \left( {n + 1} \right)} \right\|^{2} ,l = L \\ \end{aligned} $$
(B.6)

are satisfied, the sufficient condition is obtained. According to Inequations (B.3)–(B.6), one can obtain following inequations:

$$ \left\{ {\begin{array}{*{20}l} \begin{aligned} \left( {1 - a^{\left( l \right)} { + }\sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } } \right)^{2} \le 1 - \left( {\sum\nolimits_{i = 1}^{{N_{\lambda }^{l + 1} }} {\left| {\lambda_{i}^{{\left( {l + 1} \right)}} } \right|\varLambda_{\text{in}}^{{\left( {l + 1} \right)}} } } \right)^{2} \hfill \\ \begin{array}{*{20}c} {\begin{array}{*{20}c} {\begin{array}{*{20}c} {} & {} & {} \\ \end{array} } & {} & {} & {} \\ \end{array} } & {} & {} & {} \\ \end{array} ,l{ = }1,2, \cdots L - 1 \hfill \\ \end{aligned} \hfill \\ {\left( {1 - a^{\left( l \right)} { + }\sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } } \right)^{2} \le 1,l{ = }L} \hfill \\ \end{array} } \right.. $$
(B.7)

Inequation (B.7) develops to the following:

$$ \left\{ {\begin{array}{*{20}l} {0 < \varLambda^{\left( l \right)} < 1,l{ = }1,2, \cdots L - 1} \hfill \\ {0 < \varLambda^{\left( l \right)} < {{a^{\left( l \right)} } \mathord{\left/ {\vphantom {{a^{\left( l \right)} } {\sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } }}} \right. \kern-0pt} {\sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } }},l{ = }L} \hfill \\ {\varLambda_{\text{in}}^{{\left( {l + 1} \right)}} \le {{\sqrt {1 - \left( {1 - a^{\left( l \right)} { + }\sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } } \right)^{2} } } \mathord{\left/ {\vphantom {{\sqrt {1 - \left( {1 - a^{\left( l \right)} { + }\sum\nolimits_{i = 1}^{{N_{\lambda }^{l} }} {\left| {\lambda_{i}^{\left( l \right)} } \right|\varLambda^{\left( l \right)} } } \right)^{2} } } {a^{{\left( {l + 1} \right)}} }}} \right. \kern-0pt} {a^{{\left( {l + 1} \right)}} }},l{ = }1,2, \cdots L - 1} \hfill \\ \end{array} } \right.. $$
(B.8)

Then, the proof process is completed.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

LIAO, Y., LI, H. Deep echo state network with reservoirs of multiple activation functions for time-series prediction. Sādhanā 44, 146 (2019). https://doi.org/10.1007/s12046-019-1124-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s12046-019-1124-y

Keywords

Search

Navigation