Echo State Property of Deep Reservoir Computing Networks

Abstract

In the last years, the Reservoir Computing (RC) framework has emerged as a state of-the-art approach for efficient learning in temporal domains. Recently, within the RC context, deep Echo State Network (ESN) models have been proposed. Being composed of a stack of multiple non-linear reservoir layers, deep ESNs potentially allow to exploit the advantages of a hierarchical temporal feature representation at different levels of abstraction, at the same time preserving the training efficiency typical of the RC methodology. In this paper, we generalize to the case of deep architectures the fundamental RC conditions related to the Echo State Property (ESP), based on the study of stability and contractivity of the resulting dynamical system. Besides providing a necessary condition and a sufficient condition for the ESP of layered RC networks, the results of our analysis provide also insights on the nature of the state dynamics in hierarchically organized recurrent models. In particular, we find out that by adding layers to a deep reservoir architecture, the regime of network’s dynamics can only be driven towards (equally or) less stable behaviors. Moreover, our investigation shows the intrinsic ability of temporal dynamics differentiation at the different levels in a deep recurrent architecture, with higher layers in the stack characterized by less contractive dynamics. Such theoretical insights are further supported by experimental results that show the effect of layering in terms of a progressively increased short-term memory capacity of the recurrent models.

Appendices

Appendix A: Proof of Lemma 2

Let us consider the Jacobian of deepESN state transition function in Eq. 1 evaluated at u(t) and x(t − 1). From Eq. 8 we can easily see that for every j > i, we have that $\mathbf {J}_{F^{(i)},\mathbf {x}^{(j)}}(\mathbf {u}(t),\mathbf {x}(t-1))$ is a zero matrix, as the hierarchical structure of the deepESN architecture (see Fig. 1 and Eqs. 2 and 3) tells us that state update at layer i does not depend on the previous state of the system at higher layers in the stack (i.e. at layers j, with j > i). Thereby, we can notice that J _F,x (u(t),x(t − 1)) has the structure of a lower-triangular block matrix. As such, the eigenvalues of J _F,x (u(t),x(t − 1)) are the eigenvalues of the matrices on its block diagonal, i.e. the eigenvalues of $\mathbf {J}_{F^{(i)},\mathbf {x}^{(i)}}(\mathbf {u}(t),\mathbf {x}(t-1))$ for every i = 1,2,…,N _L . Accordingly, we have that the spectral radius of J _F,x (u(t),x(t − 1)) is the maximum among the spectral radii of its diagonal blocks, i.e.,

$$ \rho(\mathbf{J}_{F,\mathbf{x}}(\mathbf{u}(t),\mathbf{x}(t-1))) = \max\limits_{k = 1,2,\ldots,N_{L}} {}\left( \rho(\mathbf{J}_{F^{(k)},\mathbf{x}^{(k)}}(\mathbf{u}(t),\mathbf{x}(t-1)))\right. $$

(25)

With the aim of computing the diagonal block matrices $\mathbf {J}_{F^{(k)},\mathbf {x}^{(k)}}(\mathbf {u}(t),\mathbf {x}(t-1))$, we observe that from Eq. 8 we have that

$$ \begin{array}{l} \mathbf{J}_{F^{(k)},\mathbf{x}^{(k)}}(\mathbf{u}(t),\mathbf{x}(t-1)) = \frac{\partial F^{(k)}(\mathbf{u}(t), \mathbf{x}^{(1)}(t-1),\ldots,\mathbf{x}^{(k)}(t-1))}{\partial \mathbf{x}^{(k)}(t-1)} = \\\\ \frac{\partial}{\partial \mathbf{x}^{(k)}(t-1)}\left( (1-a^{(k)}) \mathbf{x}^{(k)}(t-1) + a^{(k)} \tanh(\mathbf{W}_{in}^{(k)} F^{(k-1)}(\mathbf{u}(t),\mathbf{x}^{(1)}(t-1),\ldots,\mathbf{x}^{(k-1)}(t-1)) + \right.\\\\ \left. \boldsymbol{\theta}^{(k)} + \hat{\mathbf{W}}^{(k)} \mathbf{x}^{(k)}(t-1)) \right)= \\\\ (1-a^{(k)})\mathbf{I} + a^{(k)} \; \left( \begin{array}{llll} 1-(\tilde{x}_1^{(k)}(t))^2 &\; 0 &\; {\ldots} & \;0\\ 0 & \; 1-(\tilde{x}_2^{(k)}(t))^2 & \; {\ldots} & \; 0\\ {\vdots} & \; {\vdots} & \; {\ddots} & \; {\vdots} \\ 0 & \; 0 & \; {\ldots} & \; 1-(\tilde{x}_{N_R}^{(k)}(t))^2\\ \end{array} \right) \hat{\mathbf{W}}^{(k)} \end{array} $$

(26)

where for j = 1,2,…,N _R , $\tilde {x}_{j}^{(k)}(t)$ are the elements of $\tilde {\mathbf {x}}^{(k)}(t) = \tanh (\mathbf {W}_{in}^{(k)} F^{(k-1)}(\mathbf {u}(t),\mathbf {x}^{(1)}(t-1),\mathbf {x}^{(2)}(t-1),\ldots ,\mathbf {x}^{(k-1)}(t-1)) + \boldsymbol {\theta }^{(k)} + \hat {\mathbf {W}}^{(k)} \mathbf {x}^{(k)}(t-1))$.

Considering zero input and state, from Eq. 26, we can derive that for every k = 1,2,…,N _L

$$ \rho(\mathbf{J}_{F^{(k)},\mathbf{x}^{(k)}}(\mathbf{0}_{u},\mathbf{0})) = \rho ((1-a^{(k)}) \mathbf{I} + a^{(k)} \hat{\mathbf{W}}^{(k)} ) $$

(27)

and therefore Eq. 25 becomes

$$ \rho(\mathbf{J}_{F,\mathbf{x}}(\mathbf{0}_{u},\mathbf{0})) = \max\limits_{k = 1,2,\ldots,N_{L}} \rho ((1-a^{(k)}) \mathbf{I} + a^{(k)} \hat{\mathbf{W}}^{(k)} ). $$

(28)

Appendix B: Proof of Lemma 3

case (a): This case follows from the case of contractivity in standard shallow ESNs [10]. Indeed, $\forall \mathbf {u} \in \mathbb {R}^{N_U}$ and $\forall \mathbf {x}^{(1)}, \mathbf {x^{\prime }}^{(1)} \in \mathbb {R}^{N_R}$

$$\begin{array}{@{}rcl@{}} &&\|F^{(1)}(\mathbf{u},\mathbf{x}^{(1)}) - F^{(1)}(\mathbf{u},\mathbf{x}^{\prime(1)})\|\\ &=&\|(1 - a^{(1)}) \mathbf{x}^{(1)} + a^{(1)} \tanh\left( \mathbf{W}_{in}^{(1)} \mathbf{u} + \boldsymbol{\theta}^{(1)} + \hat{\mathbf{W}}^{(1)} \mathbf{x}^{(1)}\right) \\ &&-(1 - a^{(1)}) \mathbf{x}^{\prime(1)} -a^{(1)}\tanh\left( \mathbf{W}_{in}^{(1)} \mathbf{u} + \boldsymbol{\theta}^{(1)} + \hat{\mathbf{W}}^{(1)} \mathbf{x}^{\prime(1)}\right)\| \end{array} $$

$$\begin{array}{@{}rcl@{}} &=&\|(1 \,-\, a^{(1)}) (\mathbf{x}^{(1)}\,-\,\mathbf{x}^{\prime(1)}) \,+\, a^{(1)} (\tanh\left( \mathbf{W}_{in}^{(1)} \mathbf{u} + \boldsymbol{\theta}^{(1)} \!+ \hat{\mathbf{W}}^{(1)} \mathbf{x}^{(1)}\right) \\ &&-\tanh\left( \mathbf{W}_{in}^{(1)} \mathbf{u} + \boldsymbol{\theta}^{(1)} + \hat{\mathbf{W}}^{(1)} \mathbf{x}^{\prime(1)}\right))\| \\ &\leq&(1 - a^{(1)}) \|\mathbf{x}^{(1)} - \mathbf{x}^{\prime(1)}\| + a^{(1)} \|\mathbf{W}_{in}^{(1)}\mathbf{u}+\boldsymbol{\theta}^{(1)}+\hat{\mathbf{W}}^{(1)}\mathbf{x}^{(1)} \\ &&-\mathbf{W}_{in}^{(1)}\mathbf{u} - \boldsymbol{\theta}^{(1)} - \hat{\mathbf{W}}^{(1)}\mathbf{x}^{\prime(1)}\| \\ &\leq&(1 - a^{(1)}) \|\mathbf{x}^{(1)}- \mathbf{x}^{\prime(1)}\| + a^{(1)} \|\hat{\mathbf{W}}^{(1)}\| \|\mathbf{x}^{(1)} - \mathbf{x}^{\prime(1)}\| \\ &=&((1 - a^{(1)}) + a^{(1)} \|\hat{\mathbf{W}}^{(1)}\|) \|\mathbf{x}^{(1)} - \mathbf{x}^{\prime(1)}\| \end{array} $$

(29)

from which it follows that $C^{(1)} = (1 - a^{(1)}) + a^{(1)} \|\hat {\mathbf {W}}^{(1)}\|$ is a Lipschitz constant for F ⁽¹⁾. Thus if C ⁽¹⁾ < 1 then F ⁽¹⁾ is a contraction (see Definition 2).

case (b): In this case, assuming F ⁽ⁱ⁻¹⁾ is a contraction with a Lipschitz constant C ⁽ⁱ⁻¹⁾ < 1, $\forall \mathbf {u} \in \mathbb {R}^{N_U}$ and $\forall \mathbf {x}^{(1)}, \ldots , \mathbf {x}^{(i)}, \mathbf {x}^{\prime (1)}, \ldots , \mathbf {x}^{\prime (i)} \in \mathbb {R}^{N_R}$

$$\begin{array}{@{}rcl@{}} \begin{array}{l} \|F^{(i)}(\mathbf{u}, \mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(i)}) - F^{(i)}(\mathbf{u}, \mathbf{x}'^{(1)},\ldots, \mathbf{x}'^{(i)})\| = \\ \\ \| (1-a^{(i)}) \mathbf{x}^{(i)} + a^{(i)} \tanh (\mathbf{W}_{in}^{(i)}F^{(i-1)}(\mathbf{u}, \mathbf{x}^{(1)},\ldots, \mathbf{x}^{(i-1)}) + \boldsymbol{\theta}^{(i)} + \hat{\mathbf{W}}^{(i)} \mathbf{x}^{(i)}) - \\ \;\;(1-a^{(i)}) \mathbf{x}'^{(i)} - a^{(i)} \tanh (\mathbf{W}_{in}^{(i)}F^{(i-1)}(\mathbf{u}, \mathbf{x}'^{(1)}, \ldots, \mathbf{x}'^{(i-1)}) + \boldsymbol{\theta}^{(i)} + \hat{\mathbf{W}}^{(i)} \mathbf{x}'^{(i)}) \| = \\ \\ \| (1-a^{(i)}) (\mathbf{x}^{(i)} - \mathbf{x}'^{(i)}) + a^{(i)} (\tanh (\mathbf{W}_{in}^{(i)}F^{(i-1)}(\mathbf{u}, \mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(i-1)}) + \boldsymbol{\theta}^{(i)} + \hat{\mathbf{W}}^{(i)} \mathbf{x}^{(i)}) - \\ \;\; \tanh (\mathbf{W}_{in}^{(i)}F^{(i-1)}(\mathbf{u}, \mathbf{x}'^{(1)}, \ldots, \mathbf{x}'^{(i-1)}) + \boldsymbol{\theta}^{(i)} + \hat{\mathbf{W}}^{(i)} \mathbf{x}'^{(i)})) \| \leq \\ \\ (1-a^{(i)}) \|\mathbf{x}^{(i)} - \mathbf{x}'^{(i)}\| + a^{(i)} \|\mathbf{W}_{in}^{(i)}F^{(i-1)}(\mathbf{u}, \mathbf{x}^{(1)},\ldots, \mathbf{x}^{(i-1)}) + \boldsymbol{\theta}^{(i)} + \hat{\mathbf{W}}^{(i)} \mathbf{x}^{(i)} - \\ \mathbf{W}_{in}^{(i)}F^{(i-1)}(\mathbf{u}, \mathbf{x}'^{(1)},\ldots,\mathbf{x}'^{(i-1)}) - \boldsymbol{\theta}^{(i)} - \hat{\mathbf{W}}^{(i)} \mathbf{x}'^{(i)}\| = \\ \\ (1-a^{(i)}) \|\mathbf{x}^{(i)} - \mathbf{x}'^{(i)}\| + a^{(i)} \| \mathbf{W}_{in}^{(i)} (F^{(i-1)}(\mathbf{u}, \mathbf{x}^{(1)},\ldots, \mathbf{x}^{(i-1)}) - \\ F^{(i-1)}(\mathbf{u}, \mathbf{x}'^{(1)},\ldots, \mathbf{x}'^{(i-1)})) + \hat{\mathbf{W}}^{(i)} (\mathbf{x}^{(i)} - \mathbf{x}'^{(i)})\| \leq \\ \\ (1-a^{(i)}) \|\mathbf{x}^{(i)} - \mathbf{x}'^{(i)}\| + a^{(i)} \left( \| \mathbf{W}_{in}^{(i)} \| \| F^{(i-1)}(\mathbf{u}, \mathbf{x}^{(1)},\ldots, \mathbf{x}^{(i-1)}) - \right.\\ \left.F^{(i-1)}(\mathbf{u}, \mathbf{x}'^{(1)}, \ldots, \mathbf{x}'^{(i-1)}) \| + \|\hat{\mathbf{W}}^{(i)}\| \|\mathbf{x}^{(i)} - \mathbf{x}'^{(i)}\| \right) \leq \\ \\ (1-a^{(i)}) \|\mathbf{x}^{(i)} - \mathbf{x}'^{(i)}\| + a^{(i)} \left( \| \mathbf{W}_{in}^{(i)}\| \, C^{(i-1)} \, \|(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(i-1)})-\right.\\ \left.(\mathbf{x}'^{(1)}, \ldots, \mathbf{x}'^{(i-1)})\| + \|\hat{\mathbf{W}}^{(i)}\| \|\mathbf{x}^{(i)} - \mathbf{x}'^{(i)} \|\right) \leq \\ \\ (1-a^{(i)}) \|(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(i)})-(\mathbf{x}'^{(1)},\ldots, \mathbf{x}'^{(i)})\| + a^{(i)} \left( C^{(i-1)} \|\mathbf{W}_{in}^{(i)}\| \|(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(i)})-\right.\\ \left.(\mathbf{x}'^{(1)},\ldots, \mathbf{x}'^{(i)})\| + \|\hat{\mathbf{W}}^{(i)}\| \|(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(i)})-(\mathbf{x}'^{(1)},\ldots, \mathbf{x}'^{(i)})\|\right) = \\ \\ (1-a^{(i)}) \|(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(i)})-(\mathbf{x}'^{(1)}, \ldots, \mathbf{x}'^{(i)})\| + \\ a^{(i)} \left( C^{(i-1)} \|\mathbf{W}_{in}^{(i)}\| + \|\hat{\mathbf{W}}^{(i)}\|\right) \|(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(i)})-(\mathbf{x}'^{(1)},\ldots, \mathbf{x}'^{(i)})\| = \\ \\ \left[(1-a^{(i)}) + a^{(i)}\left( C^{(i-1)} \|\mathbf{W}_{in}^{(i)}\| + \|\hat{\mathbf{W}}^{(i)}\|\right)\right] \|(\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(i)})-(\mathbf{x}'^{(1)}, \ldots, \mathbf{x}'^{(i)})\| \end{array} \end{array} $$

(30)

from which we can see that $C^{(i)} = (1-a^{(i)}) + a^{(i)}\left (C^{(i-1)} \|\mathbf {W}_{in}^{(i)}\| + \|\hat {\mathbf {W}}^{(i)}\|\right )$ is a Lipschitz constant for F ⁽ⁱ⁾, and thereby whenever C ⁽ⁱ⁾ < 1 it results that F ⁽ⁱ⁾ is a contraction (see Definition 2).

Appendix C: Proof of Theorem 2

Proof

Given any input string of length N, denoted by s _N = [u(1),…,u(N)], and for every couple of deepESN global states $\mathbf {x} = (\mathbf {x}^{(1)}, \ldots , \mathbf {x}^{(N_{L})}) \in \mathbb {R}^{N_{L} N_{R}}$ and $\mathbf {x}^{\prime } = (\mathbf {x}^{\prime (1)},\ldots , \mathbf {x}^{\prime (N_{L})}) \in \mathbb {R}^{N_{L} N_{R}}$, we have that:

$$\begin{array}{@{}rcl@{}} &&\|\hat{F}(\mathbf{s}_{N},\mathbf{x}) - \hat{F}(\mathbf{s}_{N},\mathbf{x}^{\prime})\| \\ &=&\|\hat{F}([\mathbf{u}(1),\ldots, \mathbf{u}(N)],\mathbf{x}) - \hat{F}([\mathbf{u}(1), \ldots, \mathbf{u}(N)],\mathbf{x}^{\prime})\| \\ &=&\|F(\mathbf{u}(N),\hat{F}([\mathbf{u}(1), \ldots, \mathbf{u}(N-1)],\mathbf{x})) \\ &&-F(\mathbf{u}(N),\hat{F}([\mathbf{u}(1),\ldots, \mathbf{u}(N-1)],\mathbf{x}^{\prime}))\| \\ &\leq& C \|\hat{F}([\mathbf{u}(1),\ldots, \mathbf{u}(N\,-\,1)],\mathbf{x}) \,-\, \hat{F}([\mathbf{u}(1),\ldots, \mathbf{u}(N\,-\,1)],\mathbf{x}^{\prime})\| \\ &=&C \|F(\mathbf{u}(N-1),\hat{F}([\mathbf{u}(1),\ldots, \mathbf{u}(N-2)],\mathbf{x})) \\ &&- F(\mathbf{u}(N-1),\hat{F}([\mathbf{u}(1),\ldots, \mathbf{u}(N-2)],\mathbf{x}^{\prime}))\| \\ &\leq& C^{2} \|\hat{F}([\mathbf{u}(1), \ldots, \mathbf{u}(N\,-\,2)],\mathbf{x}) \,-\, \hat{F}([\mathbf{u}(1), \ldots, \mathbf{u}(N\,-\,2)],\mathbf{x}^{\prime})\| \\ &\leq&\ldots \\ && C^{N-1} \|\hat{F}([\mathbf{u}(1)],\mathbf{x}) - \hat{F}([\mathbf{u}(1)],\mathbf{x}^{\prime})\| \\ &=&C^{N-1} \|F(\mathbf{u}(1),\hat{F}([\;],\mathbf{x})) - F(\mathbf{u}(1),\hat{F}([\;],\mathbf{x}^{\prime}))\| \\ &=&C^{N-1} \|F(\mathbf{u}(1),\mathbf{x}) - F(\mathbf{u}(1),\mathbf{x}^{\prime})\| \\ &\leq& C^{N} \, \|\mathbf{x} - \mathbf{x}^{\prime}\| \\ &=&C^{N} \, \max\limits_{k = 1,2,\ldots,N_{L}} \|\mathbf{x}^{(k)} - \mathbf{x}^{\prime(k)} \| \\ &\leq& C^{N} D \end{array} $$

(31)

from which it follows that $\|\hat {F}(\mathbf {s}_{N},\mathbf {x}) - \hat {F}(\mathbf {s}_{N},\mathbf {x}^{\prime })\|$ is upper bounded by a term that approaches 0 as N →∞. Thereby the ESP condition in Definition 1 holds. □