Adaptive online sequential extreme learning machine for dynamic modeling

Abstract

Extreme learning machine (ELM) is an emerging machine learning algorithm for training single-hidden-layer feedforward networks (SLFNs). The salient features of ELM are that its hidden layer parameters can be generated randomly, and only the corresponding output weights are determined analytically through the least-square manner, so it is easier to be implemented with faster learning speed and better generalization performance. As the online version of ELM, online sequential ELM (OS-ELM) can deal with the sequentially coming data one by one or chunk by chunk with fixed or varying chunk size. However, OS-ELM cannot function well in dealing with dynamic modeling problems due to the data saturation problem. In order to tackle this issue, in this paper, we propose a novel OS-ELM, named adaptive OS-ELM (AOS-ELM), for enhancing the generalization performance and dynamic tracking capability of OS-ELM for modeling problems in nonstationary environments. The proposed AOS-ELM can efficiently reduce the negative effects of the data saturation problem, in which approximate linear dependence (ALD) and a modified hybrid forgetting mechanism (HFM) are adopted to filter the useless new data and alleviate the impacts of the outdated data, respectively. The performance of AOS-ELM is verified using selected benchmark datasets and a real-world application, i.e., device-free localization (DFL), by comparing it with classic ELM, OS-ELM, FOS-ELM, and DU-OS-ELM. Experimental results demonstrate that AOS-ELM can achieve better performance.

Appendices

Appendix A

Proof of Theorem 1

Assuming that from time q to time $q+r-1$, there are input vector ${X_q}$ and output vector ${Y_q}$ of the r samples, where

$$\begin{aligned} {X_{q + r - 1}}= & {} [{x_q},{x_{q + 1}},\ldots ,{x_{q + r - 1}}] \end{aligned}$$

(29)

$$\begin{aligned} {Y_{q + r - 1}}= & {} [{y_q},{y_{q + 1}},\ldots ,{y_{q + r - 1}}] \end{aligned}$$

(30)

The corresponding output weights ${\beta _{q + r - 1}}$ can be obtained by

$$\begin{aligned} Min:\left\| {{H_{q + r - 1}}{\beta _{q + r - 1}} - {Y_{q + r - 1}}} \right\| \end{aligned}$$

(31)

where ${H_{q + r - 1}}$ is the hidden layer output matrix of the r samples from time q to time $q+r$-1:

$$\begin{aligned} {H_{q + r - 1}}= & {} \left[ {\begin{array}{*{20}{c}} {G({a_1},{b_1},{x_q})}&{} \cdots &{}{G({a_N},{b_N},{x_q})}\\ \vdots &{} \ddots &{} \vdots \\ {G({a_1},{b_1},{x_{q + r - 1}})}&{} \cdots &{}{G({a_N},{b_N},{x_{q + r - 1}})} \end{array}} \right] \end{aligned}$$

(32)

Let ${P_{q + r - 1}} = K_{q + r - 1}^{ - 1}$, the solution of ${\beta _{q + r - 1}}$ is

$$\begin{aligned} {\beta _{q + r - 1}} = {K_{q + r - 1}^{ - 1}}H_{q + r - 1}^T{Y_{q + r - 1}} \end{aligned}$$

(33)

where ${K_{q + r - 1}} = H_{q + r - 1}^T{H_{q + r - 1}}$.

At time $q+r$, when a data pair $({x_{q + r}},{y_{q + r}})$ comes, ${H_{q + r - 1}}$ becomes

$$\begin{aligned} \begin{array}{l} {H_{q + r}} = \left[ {\begin{array}{*{20}{c}} {G({a_1},{b_1},{x_q})}&{} \cdots &{}{G({a_N},{b_N},{x_q})}\\ \vdots &{} \ddots &{} \vdots \\ {G({a_1},{b_1},{x_{q + r - 1}})}&{} \cdots &{}{G({a_N},{b_N},{x_{q + r - 1}})}\\ {G({a_1},{b_1},{x_{q + r}})}&{} \cdots &{}{G({a_N},{b_N},{x_{q + r}})} \end{array}} \right] \nonumber \\ ~~~~~~~= \left[ {\begin{array}{*{20}{c}} {{H_{q + r - 1}}}\\ {{h^T}(q + r)} \end{array}} \right] \end{array}\nonumber \\ \end{aligned}$$

(34)

The corresponding output weights ${\beta _{q + r}}$ can be obtained by

$$\begin{aligned} Min:\left\| {{H_{q + r}}{\beta _{q + r}} - {Y_{q + r}}} \right\| \end{aligned}$$

(35)

where ${Y_{q + r}} = [{y_q},{y_{q + 1}},\ldots ,{y_{q + r - 1}},{y_{q + r}}]$.

The solution of ${\beta _{q + r}}$ is

$$\begin{aligned} {\beta _{q + r}} = {K_{q + r - 1}^{ - 1}}H_{q + r}^T{Y_{q + r}} \end{aligned}$$

(36)

where

$$\begin{aligned} \begin{aligned} {K_{q + r}}&= H_{q + r}^T{H_{q + r}}\\&= [\begin{array}{*{20}{c}} {H_{q + r - 1}^T}&{h(q + r)} \end{array}]\left[ {\begin{array}{*{20}{c}} {{H_{q + r - 1}}}\\ {{h^T}(q + r)} \end{array}} \right] \\&= H_{q + r - 1}^T{H_{q + r - 1}} + h(q + r){h^T}(q + r)\\&= {[{K_{q + r - 1}} + h(q + r){h^T}(q + r)]^{ - 1}} \end{aligned} \end{aligned}$$

(37)

Then,

$$\begin{aligned} \begin{aligned} {\beta _{q + r}}&= K_{q + r}^{ - 1}H_{q + r}^T{Y_{q + r}}\\&= K_{q + r}^{ - 1}\sum \limits _{i = q}^{q + r} {h(i)t(i)} \\&= K_{q + r}^{ - 1}[H_{q + r - 1}^T{Y_{q + r - 1}} + h(q + r)y(q + r)]\\&= K_{q + r}^{ - 1}[{K_{q + r - 1}}{\beta _{q + r - 1}} + h(q + r)y(q + r)]\\&= K_{q + r}^{ - 1}\left\{ {[{K_{q + r}} - h(q + r){h^T}(q + r)]{\beta _{q + r - 1}}}\right. \\&\quad \left. { + h(q + r)y(q + r)} \right\} \\&= {\beta _{q + r - 1}} + K_{q + r}^{ - 1}h(q + r)\\&\qquad [y(q + r) - {h^T}(q + r){\beta _{q + r - 1}}] \end{aligned} \end{aligned}$$

(38)

Accordingly, we have

$$\begin{aligned} {P_{q \!+ \!r}}&= {P_{q \!+\! r \!- \!1}}\! -\! {P_{q \!+\! r\! - \!1}}h(q \!+ \!r)\nonumber \\&\qquad {[I\! +\! {h^T}(q \!+\! r){P_{q \!+\! r - 1}}(q \!+\! r)]^{ - 1}}{h^T}(q \!+ \!r){P_{q \!+\! r - 1}}\nonumber \\&= [I - \frac{{{P_{q + r - 1}}h(q + r){h^T}(q + r)}}{{1 + {h^T}(q + r){P_{q + r - 1}}h(q + r)}}]{P_{q + r - 1}} \end{aligned}$$

(39)

Because ${P_{q + r - 1}} = {(H_{q + r - 1}^T{H_{q + r - 1}})^{ - 1}}$ and ${P_{q + r}} = {P_{q + r}} + h(q + r){h^T}(q + r)$, ${P_{q + r - 1}}$ and ${P_{q + r}}$ are positive definite matrices. Thus, we have

$$\begin{aligned} {P_{q + r}} < {P_{q + r - 1}} \end{aligned}$$

(40)

According to (40), we have the following relationship:

$$\begin{aligned} \mathop {\lim }\limits _{(q + r) \rightarrow \infty } {P_{q + r}} = 0 \end{aligned}$$

(41)

To sum up, extremely, OS-ELM will lose its correction capability. $\square$

Appendix B

Detailed derivation of ALD:

Let ${X_{{N_0}}} = [{x_1},{x_2},\ldots ,{x_{{N_0}}}]$, and the mean of each variable of the initial training data ${\aleph _0} = \{ ({x_i},{y_i})\} _{i = 1}^{{N_0}}$ is

$$\begin{aligned} {u_{{\aleph _0}}}= & {} \frac{1}{{{N_0}}}{({X_{{N_0}}})^T} \cdot {\mathbf{{1}}_{{N_0}}} \end{aligned}$$

(42)

where ${\mathbf{{1}}_{{N_0}}} = {[1,1,\ldots ,1]^T}$. The data scaled to zero mean and unit variance can be represented as

$$\begin{aligned} {\tilde{X}_{{N_0}}}= & {} ({X_{{N_0}}} - {\mathbf{{1}}_{{N_0}}}u_{{\aleph _0}}^T) \cdot \sum \nolimits _{{N_0}}^{ - 1} {} \end{aligned}$$

(43)

where $\sum \nolimits _{{N_0}}^{} { = diag({\sigma _{{N_0}1}},{\sigma _{{N_0}2}},\ldots ,{\sigma _{{N_0}w}})}$, and ${\sigma _{{N_0}w}}$ stands for the standard deviation of the wth datum.

When the new datum ${x_{k + 1}}$ arrives, the corresponding mean and the standard deviation can be calculated by

$$\begin{aligned} {u_{{N_0} + 1}}= & {} \frac{{{N_0}}}{{{N_0} + 1}}{u_{{N_0}}} + \frac{1}{{{N_0} + 1}}{({x_{{N_0} + 1}})^T} \end{aligned}$$

(44)

$$\begin{aligned} \sigma _{\left( {{N_0} + 1} \right) i}^2 \!= & {} \!\frac{{{N_0} - 1}}{{{N_0}}}\sigma _{{N_0}i}^2 + \varDelta u_{{N_0} + 1}^2(i)\nonumber \\&\quad \!+\! \frac{1}{{{N_0}}}{\left\| {{x_{{N_0} + 1}}(i) - {u_{{N_0} + 1}}(i)} \right\| ^2} \end{aligned}$$

(45)

Let ${x_{{N_0} + 1}}$ be scaled with

$$\begin{aligned} {\tilde{x}_{{N_0} + 1}}= & {} ({x_{{N_0} + 1}} - \mathbf{{1}} \cdot u_{{N_0} + 1}^T)\sum \nolimits _{{N_0} + 1}^{ - 1} {} \end{aligned}$$

(46)

where $\sum \nolimits _{{N_0} + 1}^{} { = diag({\sigma _{({N_0} + 1)1}},{\sigma _{({N_0} + 1)2}},\ldots ,{\sigma _{({N_0} + 1)w}})}$.

Expanding the ALD shown in (18), we have

$$\begin{aligned} \begin{aligned} {\delta _{k + 1}}&= \min \left\{ {\sum \limits _{l,m = 1}^{{N_0}} {{\alpha _l}{\alpha _m}\left\langle {{{\tilde{x}}_l},{{\tilde{x}}_m}} \right\rangle \! - \! 2\sum \limits _{m = 1}^{{N_0}} {{\alpha _m}\left\langle {{{\tilde{x}}_m},{{\tilde{x}}_{{N_0} + 1}}} \right\rangle } }}\right. \\&\left. \qquad {{{ \! + \!\left\langle {{{\tilde{x}}_m},{{\tilde{x}}_{{N_0} + 1}}} \right\rangle } } } \right\} \\&= \min \left\{ {\alpha _{{N_0} + 1}^T{J_{{N_0}}}{\alpha _{{N_0} + 1}} - 2\alpha _{{N_0} + 1}^T{j_{{N_0}}} + {{\tilde{j}}_{{N_0} + 1}}} \right\} \end{aligned} \end{aligned}$$

(47)

where ${J_{{N_0}}} = {\tilde{X}_{{N_0}}} \cdot \tilde{X}_{{N_0}}^T$, ${j_{{N_0}}} = {\tilde{X}_{{N_0}}} \cdot \tilde{x}_{{N_0} + 1}^T$, ${\tilde{j}_{{N_0} + 1}} = {\tilde{x}_{{N_0} + 1}} \cdot \tilde{x}_{{N_0} + 1}^T$.

In order to minimize ${\delta _{k + 1}}$, we have

$$\begin{aligned} {\alpha _{{N_0} + 1}} = J_{{N_0}}^{ - 1} \cdot {j_{{N_0}}} \end{aligned}$$

(48)

Substituting (48) into (47), we can obtain the recursive ALD:

$$\begin{aligned} {\delta _{k + 1}}= & {} {\tilde{j}_{{N_0} + 1}} - j_{{N_0}}^T{\alpha _{{N_0} + 1}} = {\tilde{j}_{{N_0} + 1}} - j_{{N_0}}^TJ_{{N_0}}^{ - 1} \cdot {j_{{N_0}}} \end{aligned}$$

(49)

Thus, we have

$$\begin{aligned} {\tilde{x}_{k + 1}}= & {} \sum \limits _{l = 1}^{{N_0}} {{\alpha _l}} {\tilde{x}_l} + \varepsilon \end{aligned}$$

(50)

where $\xi$ is the error.