Self-attention-based time-variant neural networks for multi-step time series forecasting

Abstract

Time series forecasting is ubiquitous in various scientific and industrial domains. Powered by recurrent and convolutional and self-attention mechanism, deep learning exhibits high efficacy in time series forecasting. However, the existing forecasting methods are suffering some limitations. For example, recurrent neural networks are limited by the gradient vanishing problem, convolutional neural networks cost more parameters, and self-attention has a defect in capturing local dependencies. What’s more, they all rely on time invariant or stationary since they leverage parameter sharing by repeating a set of fixed architectures with fixed parameters over time or space. To address the above issues, in this paper we propose a novel time-variant framework named Self-Attention-based Time-Variant Neural Networks (SATVNN), generally capable of capturing dynamic changes of time series on different scales more accurately with its time-variant structure and consisting of self-attention blocks that seek to better capture the dynamic changes of recent data, with the help of Gaussian distribution, Laplace distribution and a novel Cauchy distribution, respectively. SATVNN obviously outperforms the classical time series prediction methods and the state-of-the-art deep learning models on lots of widely used real-world datasets.

Appendix

Proof of convergence of SATVNN

The weight of the time-invariant network is often constant, and the mapping between the input and output of the network is fixed after training. However, the mapping relationship between the input and output of the time-variant system will change with time. In this paper, we propose a SATVNN with time-variant weights, which is used to establish a neural network each time, and the neural network at that time is used to approach the mapping relationship between the input and output of the system at that time. We regard time-variant weight learning as an unknown time-variant parameter estimation problem for nonlinear time-variant systems. In [57], an iterative learning least squares algorithm is proposed to train the weights of the time-variant network. Theoretical analysis shows that the iterative learning least squares algorithm can ensure that the modeling error converges to zero along the iteration axis in the whole time interval. We will elaborate on the reasons in the following paragraphs.

Time-variant neural network The time-variant neural network whose input, output and weight change with time can be expressed as follows:

$$\begin{aligned} y(t)={\mathcal {E}}^{\mathrm{T}}(x(t)) {W}(t) \end{aligned}$$

(31)

or

$$\begin{aligned} y(t)={\mathcal {E}}^{\mathrm{T}}(x(t),t) {W}(t) \end{aligned}$$

(32)

where $x(t)=\left[ x_{1}(t), x_{2}(t), \ldots , x_{n}(t)\right] ^{\mathrm{T}} \in {\mathbb{R}}^{n}$ is input vector. ${\mathcal {E}}(x(t)) = [{\mathcal {E}}_{1}(x(t)), {\mathcal {E}}_{2}(x(t)), \ldots , {\mathcal {E}}_{l}(x(t))]^{\mathrm{T}}$ is a set of activation function vectors. y(t) is output. ${W}(t)=\left[ w_{1}(t), \ldots , w_{l}(t)\right] ^{\mathrm{T}}$ is an adjustable time-variant weight vector. l is the number of neuron nodes.

Time-variant system Let the nonlinear discrete time-variant system be:

$$\begin{aligned} y_{m}(t)=f\left( x_{m}(t), t\right) \end{aligned}$$

(33)

where f is a nonlinear time-variant function, $x_{m}(t)$ is input vector. $t \in \{0,1, \ldots , N\}$ is a finite time interval. $m \in \{0,1, \ldots \}$ is the number of iterations and $y_{m}(t)$ is output. In the following discussion, we all assume that $x_{m}(t)$ is bounded.

Iterative learning least square method Let the nonlinear time-variant system (33) be run repeatedly for m time from the 0th time, and the input and output data pairs $\{\left( x_{i}(t), y_{i}(t)\right) , t=0,1, \ldots , N, i=0,1,\ldots ,m\}$ are generated by the system. We will use the time-variant neural network (32) to approximate the above input–output mapping.

First, we define ${{E}}_{m}(t)=[{\mathcal {E}}_{0}^{\mathrm{T}}(t), {\mathcal {E}}_{1}^{\mathrm{T}}(t), \ldots , {\mathcal {E}}_{m}^{\mathrm{T}}(t)]^{\mathrm{T}}$ and ${Y}_{m}(t)=[y_{0}(t), y_{1}(t), \ldots , y_{m}(t)]^{\mathrm{T}}$,where ${\mathcal {E}}_{i}(t)={\mathcal {E}} (x_{i}(t), t)$, according to Eq. (32),

$$\begin{aligned} Y_{m}(t)={E}_{m}(t) W(t) \end{aligned}$$

(34)

We hope to find out the least square solution for W(t), ${\hat{W}}_{m}(t)$, so that the following loss function is minimized.

$$\begin{aligned} \begin{aligned} J_{m}({\hat{W}}_{m}(t), t)&= \frac{1}{2}[Y_{m}(t)-{E_{m}(t)} {\hat{W}}_{m}(t)]^{\mathrm{T}} \\&\quad [{Y}_{m}(t)-{E_{m}(t)} {\hat{W}}_{m}(t)] \end{aligned} \end{aligned}$$

(35)

Write Eq. (35) further as:

$$\begin{aligned} \begin{aligned}&2 J_{m} \left( {\hat{W}}_{m}(t), t\right) ={Y}_{m}^{\mathrm{T}}(t) \left[ {I}-{{E}}_{m}(t) \left( {{E}}_{m}^{\mathrm{T}}(t) {E}_{m}\right) ^{-1}(t) {E}_{m}^{\mathrm{T}}(t) \right] \\&{Y}_{m}(t) + \left[ {\hat{W}}_{m}(t) - \left[ {{E}}_{m}^{\mathrm{T}}(t) {{E}}_{{m}}(t)\right] ^{-1} {{E}}_{m}^{\mathrm{T}}(t) {Y}_{{m}}(t)\right] ^{\mathrm{T}} \\&{E}_{m}^{\mathrm{T}}(t) {E}_{m}(t) \left[ \hat{W}_{m}(t)-\left( {{E}}_{m}^\mathrm{T}(t) {E}_{m}(t)\right) ^{-1} {{E}}_{m}^{\mathrm{T}}(t) {Y}_{m}(t+1)\right] \end{aligned} \end{aligned}$$

(36)

Note that only the second term on the right side of the above formula is related to ${\hat{W}}_{m}(t)$, set this item zero. We assume that ${{E}}_{m}^{\mathrm{T}}(t) {{E}}_{m}(t)$ is invertible and achieve the minimum value:

$$\begin{aligned} {\hat{W}}_{m}(t)=\left( {{E}}_{m}^{\mathrm{T}}(t) {{E}}_{m}(t)\right) ^{-1} {{E}}_{m}^{\mathrm{T}}(t) Y_{m}(t) \end{aligned}$$

(37)

According to Eq. (37), calculating ${\hat{W}}_{m}(t)$ requires calculating the inverse of ${E}_{m}(t)$. The dimension of ${E}_{m}(t)$ will increase, and the inverse computation will become time-consuming as iteration increases. To solve this problem, an iterative learning least square algorithm is derived.

Let us define $Q_{m}^{-1}(t)={E}_{m}^{\mathrm{T}}(t) {E}_{m}(t)$, according to the definition of ${E}_{m}(t)$, we can get:

$Q_{m+1}^{-1}(t)=\sum _{i=0}^{m+1} {\mathcal {E}}_{i}(t) {\mathcal {E}}_{i}^{\mathrm{T}}(t)=\sum _{i=0}^{m} {\mathcal {E}}_{i}(t) {\mathcal {E}}_{i}^{\mathrm{T}}(t)+{\mathcal {E}}_{m+1}(t) {\mathcal {E}}_{m+1}^{\mathrm{T}}(t)$ and namely:

$$\begin{aligned} Q_{m+1}^{-1}(t)=Q_{m}^{-1}(t)+{\mathcal {E}}_{m+1}(t) {\mathcal {E}}_{m+1}^{\mathrm{T}}(t) \end{aligned}$$

(38)

Use Matrix inversion lemma, $(A^{-1}+H R^{-1} H^{\mathrm{T}})^{-1}=A-A H({H}^{\mathrm{T}} A {H}+{R})^{-1} {H}^{T} {A}$, let ${A}={Q}_{m}(t)$, ${H}={\mathcal {E}}_{m+1}(t)$, ${R}=1$. we can get:

$$\begin{aligned} Q_{m+1}(t)=Q_{m}(t)-\frac{Q_{m}(t) {\mathcal {E}}_{m+1}(t) {\mathcal {E}}_{m+1}^{\mathrm{T}}(t) Q_{m}(t)}{1+{\mathcal {E}}_{m+1}^{\mathrm{T}}(t) Q_{m}(t) {\mathcal {E}}_{m+1}(t)} \end{aligned}$$

(39)

According to the definition of $Q_{m}^{-1}(t)$, we can compute ${\hat{W}}_{m}$ such that:

$$\begin{aligned} \begin{aligned} {\hat{W}}_{m+1}&= Q_{m+1} {E}_{m+1}^{\mathrm{T}} Y_{m+1} \\&=Q_{m+1}\left[ {E}_{m}^{\mathrm{T}} Y_{m}+{\mathcal {E}}_{m+1} y_{m+1}\right] \\&=Q_{m+1}\left[ Q_{m}^{-1} {\hat{W}}_{m}+{\mathcal {E}}_{m+1} y_{m+1}\right] \\&={\hat{W}}_{m}+Q_{m+1} {\mathcal {E}}_{m+1}\left[ y_{m+1}-{\mathcal {E}}_{m+1}^{\mathrm{T}} {\hat{W}}_{m}\right] \end{aligned} \end{aligned}$$

(40)

Define error $e_{m+1}(t)=y_{m+1}(t)-{\mathcal {E}}_{m+1}(t) {\hat{W}}_{m}(t)$

According to Eq. (39), we get:

$$\begin{aligned} {\hat{W}}_{m+1}(t)={\hat{W}}_{m}(t)+\frac{Q_{m}(t) {\mathcal {E}}_{m+1}(t)}{1+{\mathcal {E}}_{m+1}^{\mathrm{T}}(t) {Q}_{m}(t){{\mathcal {E}}_{m+1}}(t)} e_{m+1}(t) \end{aligned}$$

(41)

According to Eqs. (39) and (41), update laws for $Q_{m}(t)$ and ${{\hat{W}}}_{m}(t)$ indicate iterative learning least square algorithm of time-variant neural for network training.

Convergence analysis We analyze the convergence of iterative learning algorithm.

Theorem 1

For system (33), the learning algorithm Eqs. (39)–(41) has the following properties:

• (i) For $t=0,1, \ldots , N$ and $m=0,1, \ldots ,$

  $$\begin{aligned} \left\| {\tilde{W}}_{m+1}(t)\right\| ^{2}&\leqslant \rho _{m}(t)\left\| {\tilde{W}}_{m}(t)\right\| ^{2} \end{aligned}$$

  (42)
  $$\begin{aligned} \left\| {\tilde{W}}_{m}(t)\right\| ^{2}&\leqslant \rho _{0}(t)\left\| {\tilde{W}}_{0}(t)\right\| ^{2} \end{aligned}$$

  (43)

  where $\rho _{m}(t)={\lambda _{\max }(Q_{m}^{-1}(t))}/{\lambda _{\min }(Q_{m}^{-1}(t))}$, minimum eigenvalues and maximum eigenvalues of $Q_{m}^{-1}(t)$ are represented by $\lambda _{\min }(Q_{m}^{-1}(t))$ and $\lambda _{\max }(Q_{m}^{-1}(t))$, respectively.

• (ii) For $t=0,1, \ldots , N$,

  $$\begin{aligned} \lim _{m \rightarrow \infty } e_{m+1}(t)=0 \end{aligned}$$

  (44)

Proof

By definition, $e_{m+1}(t)={\mathcal {E}}_{m+1}^{\mathrm{T}}(t) \tilde{{W}}_{m}(t)$, according to Eqs. (39)–(41), we can get:

$$\begin{aligned}&Q_{m+1}(t) Q_{m}^{-1}(t)=I-\frac{Q_{m}(t) {\mathcal {E}}_{m+1}(t) {\mathcal {E}}_{m+1}^{\mathrm{T}}(t)}{1+{\mathcal {E}}_{m+1}^{\mathrm{T}}(t) Q_{m}(t) {\mathcal {E}}_{m+1}(t)} \end{aligned}$$

(45)

$$\begin{aligned}&{\tilde{W}}_{m+1}(t)=\left[ I-\frac{Q_{m}(t) {\mathcal {E}}_{m+1}(t) {\mathcal {E}}_{m+1}^{\mathrm{T}}(t)}{1+{\mathcal {E}}_{m+1}^{\mathrm{T}}(t) Q_{m}(t) {\mathcal {E}}_{m+1}(t)}\right] {\tilde{W}}_{m}(t) \end{aligned}$$

(46)

Combine Eqs. (45) and (46), we can get:

$$\begin{aligned} Q_{m+1}^{-1}(t) {\tilde{W}}_{m+1}(t)=Q_{m}^{-1}(t) {\tilde{W}}_{m}(t) \end{aligned}$$

(47)

Define $V_{m}(t)={\tilde{W}}_{m}^{\mathrm{T}}(t) Q_{m}^{-1}(t) {\tilde{W}}_{m}(t)$, use Eq. (45), we can get:

$$\begin{aligned} V_{m+1}(t)-V_{m}(t)=-\frac{{\tilde{W}}_{m}^{\mathrm{T}}(t) {\mathcal {E}}_{m+1}(t) {\mathcal {E}}_{m+1}^{\mathrm{T}}(t) {\tilde{W}}_{m}(t)}{1+{\mathcal {E}}_{m+1}^{\mathrm{T}}(t) Q_{m}(t) {\mathcal {E}}_{m+1}(t)} \end{aligned}$$

(48)

Therefore, $V_{m}(t)$ are non-increasing and nonnegative for m, namely

$$\begin{aligned} V_{m+1}(t) \leqslant V_{m}(t) \end{aligned}$$

(49)

According to Eq. (38), we can get:

$$\begin{aligned} \lambda _{\min }(Q_{m+1}^{-1}(t)) \geqslant \lambda _{\min }(Q_{m}^{-1}(t)) \geqslant \lambda _{\min }(Q_{0}^{-1}(t)) \end{aligned}$$

(50)

Combine Eq. (49), we can get:

$$\begin{aligned} \lambda _{\min }(Q_{m}^{-1}(t))\left\| {\tilde{W}}_{m+1}(t)\right\| ^{2} \leqslant \lambda _{\max }(Q_{m}^{-1}(t))\left\| {\tilde{W}}_{m}(t)\right\| ^{2} \end{aligned}$$

(51)

Therefore:

$$\begin{aligned} \left\| {\tilde{W}}_{m+1}(t)\right\| ^{2} \leqslant \rho _{m}(t)\left\| {\tilde{W}}_{m}(t)\right\| ^{2} \end{aligned}$$

(52)

Apparently there is $V_{m}(t) \leqslant V_{0}(t)$, combine Eq. (49), we can get:

$$\begin{aligned} \lambda _{\min }(Q_{0}^{-1}(t))\left\| {\tilde{W}}_{m}(t)\right\| ^{2} \leqslant \lambda _{\max }(Q_{0}^{-1}(t))\left\| {\tilde{W}}_{0}(t)\right\| ^{2} \end{aligned}$$

(53)

This ensures that $\left\| {\tilde{W}}_{m}(t)\right\| ^{2}$ is uniformly bounded. The property (i) is proved. $\hfill\square$

Proof

According to Eq. (48), we can get:

$$\begin{aligned} \frac{e_{m+1}^{2}(t)}{1+{\mathcal {E}}_{m+1}^{\mathrm{T}}(t) Q_{m}(t) {\mathcal {E}}_{m+1}(t)}=V_{m}(t)-V_{m+1}(t) \end{aligned}$$

(54)

Sum the above formula from 0 to m, we get:

$$\begin{aligned} \sum _{i=0}^{m} \frac{e_{i+1}^{2}(t)}{1+{\mathcal {E}}_{i+1}^{\mathrm{T}}(t) Q_{i}(t) {\mathcal {E}}_{i+1}(t)}=V_{0}(t)-V_{m+1}(t) \leqslant V_{0}(t) \end{aligned}$$

(55)

Because $V_{0}(t)$ is bounded, therefore:

$$\begin{aligned} \lim _{m \rightarrow \infty } \frac{e_{m+1}^{2}(t)}{1+{\mathcal {E}}_{m+1}^{\mathrm{T}}(t) Q_{m}(t) {\mathcal {E}}_{m+1}(t)}=0 \end{aligned}$$

(56)

Use $\lambda _{\max }\left( Q_{m+1}(t)\right) \leqslant \lambda _{\max }\left( Q_{m}(t)\right)$, we can get:

$$\begin{aligned} {\mathcal {E}}_{m+1}^{\mathrm{T}}(t) Q_{m}(t) {\mathcal {E}}_{m+1}(t) \leqslant \lambda _{\max }\left( Q_{0}(t)\right) {\mathcal {E}}_{m+1}^{\mathrm{T}}(t) {\mathcal {E}}_{m+1}(t) \end{aligned}$$

(57)

According to Eq. (56), we can get:

$$\begin{aligned} \lim _{m \rightarrow \infty } \frac{e_{m+1}(t)}{\sqrt{1+\lambda _{\max }\left( Q_{0}(t)\right) {\mathcal {E}}_{m+1}^{\mathrm{T}}(t) {\mathcal {E}}_{m+1}(t)}}=0 \end{aligned}$$

(58)

The input $x_{m}(t)$ is bounded; then, ${\mathcal {E}}_{m+1}(t)$ is bounded. In Eq. (58), the denominator term is bounded and nonzero; then, we can get:

$$\begin{aligned} \lim _{m \rightarrow \infty } e_{m+1}(t)=0, t=0,1, \ldots , N \end{aligned}$$

(59)

$\hfill\square$

This ensures the uniform convergence of the estimation error and proves the property (ii). Therefore, the learning algorithm we give ensures the boundedness of time-variant weights. The estimation error on a finite time interval can asymptotically converge to zero.