Skip to main content
Log in

Mean-square deviation analysis of the zero-attracting variable step-size LMS algorithm

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The well-known variable step-size least-mean-square (VSSLMS) algorithm provides faster convergence rate while maintaining lower mean-square error than the conventional LMS algorithm. The performance of the VSSLMS algorithm can be improved further in a channel estimation problem if the impulse response of the channel is sparse. Recently, a zero-attracting (ZA)-VSSLMS algorithm was proposed to exploit the sparsity of a channel. This was done by imposing an \(\ell _1\)-norm penalty to the original cost function of the VSSLMS algorithm which utilizes the sparsity in the filter taps during the adaptation process. In this paper, we present the mean-square deviation (MSD) analysis of the ZA-VSSLMS algorithm. A steady-state MSD expression for the ZA-VSSLMS algorithm is derived. An upper bound of the zero-attractor controller (\(\rho \)) that provides the minimum MSD is also provided. Moreover, the effect of the noise distribution on the MSD performance is shown theoretically. It is shown that the theoretical and simulation results of the algorithm are in good agreement with a wide range of parameters, different channel, input signal, and noise types.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Widrow, B., Stearns, S.D.: Adaptive Signal Processing. Prentice Hall, Eaglewood Cliffs (1985)

    MATH  Google Scholar 

  2. Haykin, S.: Adaptive Filter Theory. Prentice Hall, Upper Saddle River, NJ (2002)

  3. Gu, Y., Jin, J., Mei, S.: \(\ell _0\) norm constraint LMS algorithm for sparse system identification. IEEE Signal Process. Lett. 16, 774–777 (2009)

  4. Jin, J., Qing, Q., Gu, Y.: Robust zero-point attraction LMS algorithm on near sparse system identification. IET Signal Process. 3, 210–218 (2013)

    Article  Google Scholar 

  5. Gui, G., Peng, W., Adachi, F.: Improved adaptive sparse channel estimation based on the least mean square algorithm. IEEE Wireless Commun. Netw. Conf. (WCNC), Shanghai, China 3105–3109 (2013)

  6. Wu, F.Y., Tong, F.: Non-uniform norm constraint LMS algorithm for sparse system identification. IEEE Commun. Lett. 2, 385–388 (2014)

    Google Scholar 

  7. Chen, Y. Gu, Y., Hero, A.O.: Sparse LMS for system identification. Proceedings of the IEEE International conference on Acoustics, Speech and Signal Processing, Taipei, Taiwan, pp. 3125–3128 (2009)

  8. Zhang, S., Zhang, J.: New steady-state analysis results of variable step-size LMS algorithm with different noise distributions. IEEE Signal Process. Lett. 21, 653–657 (2014)

    Article  Google Scholar 

  9. Donoho, D.: Compressive sensing. IEEE Trans. Inf. Theory 52, 1289–1306 (2006)

    Article  MATH  Google Scholar 

  10. Guolong, Su, Jin, Jian, Yuantao, Gu, Wang, Jian: Performance analysis of \(l_0\) norm constraint least mean square algorithm. IEEE Trans. Signal Process. 60, 2223–2235 (2012)

    Article  MathSciNet  Google Scholar 

  11. Salman, M.S., Jahromi, M.N.S., Hocanin, A., Kukrer, O.: A weighted zero-attracting leaky-LMS algorithm. SoftCOM 2012 International Conference on Software, Telecommunications and Computer Networks, Croatia, pp. 1–3 (2012)

  12. Jahromi, M.N.S., Salman, M.S., Hocanin, A., Kukrer, O.: Convergence analysis of the zero-attracting variable step-size LMS algorithm for sparse system identification. Signal Image Video Process. Springer 9, 1353–1356 (2015)

    Article  Google Scholar 

  13. Kwong, R., Johnston, E.W.: A variable step size LMS algorithm. IEEE Trans. Signal Process. 40, 1633–1642 (1992)

    Article  MATH  Google Scholar 

  14. Henderson, H.V., Searle, S.R.: The vec-permutation matrix, the vec operator, and Kronecker products, a review. Linear Mulitilinear Algeb. 9(4), 271288 (1981)

    MathSciNet  MATH  Google Scholar 

  15. Alexander, G.: Kronecker Products and Matrix Calculus with Application. Halsted, New York (1981)

    MATH  Google Scholar 

  16. Papoulis, A.: Probability, Random Variables, and Stochastic Processes, 3rd edn. McGraw-Hill, NewYork (1991)

    MATH  Google Scholar 

  17. Aboulnasr, T., Mayyas, K.: A robust step-size LMS-type algorithm: analysis and simulations. IEEE Trans. Signal Process. 45(3), 631–639 (1997)

    Article  Google Scholar 

  18. Shi, K., Shi, P.: Convergence analysis of sparse LMS algorithms with l1-norm penalty based on white input signal. Signal Process. 90, 3289–3293 (2010)

    Article  MATH  Google Scholar 

  19. Diniz, P.S.R.: Adaptive Filtering: Algorithms and Practical Implementation, 4th edn. Springer, NewYork (2013)

    Book  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mohammad N. S. Jahromi.

Appendices

Appendix 1

Here, we show how we obtain mathematically the result stated in Eq. (12) from (11). By using the Kronecker product property [15], for a given arbitrary matrices A, B, and C of compatible sizes, \(vec(ABC)=(C^T\bigotimes A)vec(B)\). Then, the expression given in (11) can be transformed into the following vec(.) form and taking the expectation of both sides yields,

(18)

In steady state, we can assume the following:

  1. 1.

    \(E\left\{ e^2_{k-i-1}e^2_{k-j-1}\right\} =E\left\{ e^2_{k-i-1}\right\} E\left\{ e^2_{k-j-1}\right\} \) for \(i\ne j\) [17].

  2. 2.

    The expressions \(\left( g(i,k)\right) \), \(\left( g(i,k)\right) \left( g(i,k)\right) \), and \(\left( \varvec{\delta }(k)\varvec{\delta }^T(k)\right) \) are assumed to be independent. Hence, employing the above assumptions, we obtain (refer “Appendix 2” for proof):

    $$\begin{aligned} E\left\{ \left( g(i,k)\right) \left( g(i,k)\right) \right\} = \frac{2\alpha \left( E\left( e_\infty ^2\right) \right) ^2}{(1-\alpha )^2(1+\alpha )}+\frac{E\left( e_\infty ^4\right) }{(1-\alpha ^2)} \end{aligned}$$

    and

    $$\begin{aligned} E\left\{ g(i,k)\right\} =\frac{E\left( e_\infty ^2\right) }{1-\alpha } \end{aligned}$$

From (18), it is easy to show that the expectation of the last term is zero (\(E\{.\}=0\)). Now, let us evaluate the expectations of terms , and , individually.

Starting by term and using the independence assumption,

$$\begin{aligned}&E\left[ \lambda \gamma \varvec{\delta }(k)\mathrm{sgn}[\mathbf w ^T(k)]\left( g(i,k)\right) \right] \nonumber \\&\quad =\lambda \gamma E\left[ (\mathbf h -\mathbf w (k))\mathrm{sgn}[\mathbf w ^T(k)]\right] \times E\left[ g(i,k)\right] \end{aligned}$$
(19)

To find expectation of (19) at steady state, we need to compute \(E[\mathbf w _\infty ]\). To do so, by applying the above assumption (2), we calculate the expectation of Eq. (10) in the article as \(\lim \limits _{k\rightarrow \infty }\), we obtain

$$\begin{aligned} E[\varvec{\delta }_\infty ]= & {} \left( 1-\frac{\gamma \sigma _x^2E\left( e_\infty ^2\right) }{1-\alpha }\right) E[\varvec{\delta }_\infty ]\nonumber \\&+\frac{\lambda \gamma E\left( e_\infty ^2\right) }{1-\alpha }E[\mathrm{sgn}(\mathbf w _\infty )]\nonumber \\= & {} \frac{\lambda }{\sigma _x^2}E[\mathrm{sgn}(\mathbf w _\infty )]. \end{aligned}$$
(20)

Substituting the result of (20) in (8) as \(k\rightarrow \infty \) we get,

$$\begin{aligned} E[\mathbf w _\infty ]=\mathbf h -\frac{\lambda }{\sigma _x^2}E[\mathrm{sgn}(\mathbf w _\infty )], \end{aligned}$$
(21)

or in scalar form, (21) can be written as

$$\begin{aligned} E[w_{j,\infty }]=h_j-\frac{\lambda }{\sigma _x^2}E[\mathrm{sgn}(w_{j,\infty })], \quad j\in NZ \end{aligned}$$
(22)

where \(j\in NZ\) is the index that corresponds to a nonzero (NZ) coefficient. By [18], for sufficiently small \(\frac{\lambda }{\sigma _x^2}\), the sign of (22) is,

$$\begin{aligned} E\left[ \mathrm{sgn}\left( w_{j,\infty }\right) \right] =\mathrm{sgn}[h_j] \end{aligned}$$
(23)

And hence, using (22) and (23) gives

$$\begin{aligned} E\left[ w_{j,\infty }\right] E\left[ \mathrm{sgn}\left( w_{j,\infty }\right) \right] = \left\{ \begin{array}{ll} |h_j| &{}\quad j \in \text {NZ}\\ \frac{-\lambda }{\sigma _x^2}E^2\left[ \mathrm{sgn}(w_{j,\infty })\right] &{}\quad j \in \text {Z} \end{array}\right. \nonumber \\ \end{aligned}$$
(24)

Additionally, from (23), it is straightforward to show that for \(j\in \text {NZ}\), we have,

$$\begin{aligned} E\left[ h_j \mathrm{sgn}\left( w_{j,\infty }\right) \right] = |h_j|. \end{aligned}$$
(25)

Substituting (22), (24), (25) in (19) and letting \(E^2\left[ \mathrm{sgn}\left( w_{j,\infty }\right) \right] \) \(\approx 0\) if \(j\in \text {Z}\), the expectation in Eq. (19) at steady state reduces to,

$$\begin{aligned}&\lambda \gamma E\left[ (\mathbf h -\mathbf w (k))\mathrm{sgn}[\mathbf w ^T(k)]\right] E\left[ g(i,k)\right] \nonumber \\&\quad =\left\{ \begin{array}{ll}\frac{\gamma \lambda ^2}{(1-\alpha )\sigma _x^2}E\left( e_\infty ^2\right) &{}\quad j\in \text {NZ}\\ 0 &{}\quad j\in \text {Z} \end{array}\right. \end{aligned}$$
(26)

In similar way, it is easy to verify that the expectations of terms and in Eq. (18) are as follows,

$$\begin{aligned}&\lambda \gamma ^2E\left[ \mathbf x (k)\mathbf x ^T(k)\delta (k)\mathrm{sgn}[\mathbf w ^T(k)]g(i,k)g(i,k)\right] \nonumber \\&=\left\{ \begin{array}{ll} \frac{\gamma ^2\lambda ^2}{\sigma _x^2}\bigg [\frac{2\alpha \left( E\left( e_\infty ^2\right) \right) ^2}{(1-\alpha )^2(1+\alpha )}+\frac{E\left( e_\infty ^4\right) }{(1-\alpha ^2)}\bigg ]vec(\mathbf R )&{}\quad j\in \text {NZ}\\ 0 &{}\quad j\in \text {Z} \end{array}\right. \end{aligned}$$
(27)

and

$$\begin{aligned}&\lambda ^2\gamma ^2E\left[ \mathrm{sgn}[\mathbf w (k)]\mathrm{sgn}[\mathbf w ^T(k)]g(i,k)g(i,k)\right] \nonumber \\&=\left\{ \begin{array}{ll} \frac{\lambda ^2\gamma ^2}{N}\sum \limits _{j=0}^{N-1}\mathrm{sgn}\left[ |w_j|\right] \bigg [\frac{2\alpha \left( E\left( e_\infty ^2\right) \right) ^2}{(1-\alpha )^2(1+\alpha )}+\frac{E\left( e_\infty ^4\right) }{(1-\alpha ^2)}\bigg ]vec(\mathbf I )&{}\quad j\in \text {NZ}\\ 0 &{}\quad j\in \text {Z} \end{array}\right. \nonumber \\ \end{aligned}$$
(28)

The expectations of term and in Eq. (18) are identical to term and , respectively.

As \(k\rightarrow \infty \), let \(vec\left( \varvec{\varDelta }_\infty \right) =E\bigg [vec\bigg (\varvec{\delta }_\infty \varvec{\delta }^T_\infty )\bigg )\bigg ]\) be a vector of size \(N^2\times 1\), \(\xi =\bigg [\frac{2\alpha \left( E\left( e_\infty ^2\right) \right) ^2}{(1-\alpha )^2(1+\alpha )}+\frac{E \left( e_\infty ^4\right) }{(1-\alpha ^2)}\bigg ]\) and \(\mathbf R =E[\mathbf x _\infty \mathbf x ^T_\infty ]\). Hence, substituting the results of (26), (27) and (28) into (18) and using assumption (2) we get

$$\begin{aligned}&vec\left( \varvec{\varDelta }_\infty \right) \nonumber \\&\quad = vec(\varvec{\varDelta }_\infty )+\gamma ^2\xi E\bigg [\mathbf x _\infty \mathbf x ^T_\infty \otimes \mathbf x _\infty \mathbf x ^T_\infty \bigg ]vec\left( \varvec{\varDelta }_\infty \right) \\&\quad -\frac{\gamma E \left( e_\infty ^2\right) }{(1-\alpha )}\left( E\bigg [\mathbf I \otimes \mathbf x _\infty \mathbf x ^T_\infty \bigg ]+E\bigg [\mathbf x _\infty \mathbf x ^T_\infty \otimes \mathbf I \bigg ]\right) vec\left( \varvec{\varDelta }_\infty \right) \nonumber \\&\quad +\left[ \gamma ^2\sigma ^2_v-\frac{2\gamma ^2\lambda ^2}{\sigma _x^2}\right] \xi vec(\mathbf R )+\frac{\lambda ^2\gamma ^2}{N} \xi \sum \limits _{j=0}^{N-1}\mathrm{sgn}\left[ |w_j|\right] vec(\mathbf I ) \end{aligned}$$

which is identical to Eq. (12) in the article.

Appendix 2

Recalling that \(g(i,k)=\sum \nolimits _{i=0}^{k-2}\alpha ^i e^2(k-i-1)\), we want to evaluate the expectation of the form,

$$\begin{aligned} E\left\{ g(i,k)g(i,k)\right\} =\sum \limits _{i=0}^{k-2}\sum \limits _{j=0}^{k-2}\alpha ^{i+j}E\Bigg \{e^2(k-i-1)e^2(k-j-1)\Bigg \} \end{aligned}$$
(29)

To calculate the above expectation, we divide (29) into two cases as follows:

Case I: when \(i=j\), as \(k\rightarrow \infty \)

$$\begin{aligned}&\sum \limits _{i=0}^{k-2}\sum \limits _{j=0}^{k-2}\alpha ^{i+j}E\Bigg \{e^2(k-i-1)e^2(k-j-1)\Bigg \}\\&\quad =\frac{1}{1-\alpha ^2}E(e^{4}_{\infty }) \end{aligned}$$

Case II: when \(i\ne j\), as \(k\rightarrow \infty \) and by assumption (2) , in “Appendix 1”, we have,

$$\begin{aligned}&\sum \limits _{i=0}^{k-2}\sum \limits _{j=0}^{k-2}\alpha ^{i+j}E\Bigg \{e^2(k-i-1)e^2(k-j-1)\Bigg \}\nonumber \\&\quad =\sum _{\begin{array}{c} i=0\\ [0.5ex](i\ne j) \end{array}}^{\infty }\sum \limits _{j=0}^{\infty }\alpha ^{i}\alpha ^{j}\left( E(e^{2}_{\infty })\right) ^2 \end{aligned}$$
(30)

We note that when \(i\ne j\), the double summation in (30) reduces to:

$$\begin{aligned} \sum _{\begin{array}{c} i=0\\ (i\ne j) \end{array}}^{\infty }\sum \limits _{j=0}^{\infty }\alpha ^{i}\alpha ^{j}= & {} \left( \sum \limits _{i=0}^{\infty }\alpha ^i\right) ^2-\sum \limits _{i=0}^{\infty }(\alpha ^i)^2\\= & {} \frac{2\alpha }{(1+\alpha )(1-\alpha )^2} \end{aligned}$$

Combining the results of case (I) and (II), the expectation of (29) becomes,

$$\begin{aligned} E\left\{ g(i,k)g(i,k)\right\} =\frac{2\alpha \left( E\left( e_\infty ^2\right) \right) ^2}{(1-\alpha )^2(1+\alpha )}+\frac{E\left( e_\infty ^4\right) }{(1-\alpha ^2)} \end{aligned}$$

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jahromi, M.N.S., Salman, M.S., Hocanin, A. et al. Mean-square deviation analysis of the zero-attracting variable step-size LMS algorithm. SIViP 11, 533–540 (2017). https://doi.org/10.1007/s11760-016-0991-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-016-0991-5

Keywords

Navigation