Skip to main content
SpringerLink
Log in

Stochastic Gradient Adaptive Algorithms

  • Chapter
  • First Online:
A Rapid Introduction to Adaptive Filtering

Part of the book series: SpringerBriefs in Electrical and Computer Engineering ((BRIEFSELECTRIC))

Abstract

One way to construct adaptive algorithms leads to the so called Stochastic Gradient algorithms which will be the subject of this chapter. The most important algorithm in this family, the Least Mean Square algorithm (LMS), is obtained from the SD algorithm, employing suitable estimators of the correlation matrix and cross correlation vector. Other important algorithms as the Normalized Least Mean Square (NLMS) or the Affine Projection (APA) algorithms are obtained from straightforward generalizations of the LMS algorithm. One of the most useful properties of adaptive algorithms is the ability of tracking variations in the signals statistics. As they are implemented using stochastic signals, the update directions in these adaptive algorithms become subject to random fluctuations called gradient noise. This will lead to the question regarding the performance (in statistical terms) of these systems. In this chapter we will try to give a succinct introduction to this kind of adaptive filter and to its more relevant characteristics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The convergence analysis will be properly done and justified in Sect. 4.5.3. Here we just want to give an intuitive result concerning the limiting behavior of the LMS.

  2. 2.

    In the marginal case \(\Vert \mathbf x (n)\Vert =0\), the direction of update will be the null vector so there is no need to compute \(\mu (n)\).

  3. 3.

    The misadjustment will be properly defined in Sect. 4.5 when we analyze the convergence of adaptive filters. For now, it can be seen as the ratio between the steady state EMSE and the MMSE.

  4. 4.

    The notation \(\mathbf A ^{\dag }\) denotes the Moore-Penrose pseudoinverse of matrix \(\mathbf A \) (see Chap.  5 for further details). When \(\mathbf A =\mathbf x ^T(n)\) it can be shown that:

    $$\left[\mathbf x ^T(n)\right]^{\dag }=\frac{\mathbf x (n)}{\Vert \mathbf x (n)\Vert ^2}. $$
  5. 5.

    If this subtraction is not done properly (under the control of an adaptive filter) it might lead to an increase of the output noise power.

  6. 6.

    This is not the only possible cost function that could be considered. Another popular solution is to obtain a filter \(\mathbf w \) that completely inverts the channel \(\mathbf h \), without taking into account the noise \(v(n)\). This solution called zero forcing (ZF) [15] completely eliminates the ISI at the cost of possibly increasing the influence of the noise. However, when the noise is sufficiently small and the channel \(\mathbf h \) does not present nulls in its frequency response, ZF offers a good performance.

  7. 7.

    There are adaptive filtering variants for the equalization problem that do not require a training sequence. These are the so called blind adaptive filters [19]. Those filters do not need an exact reference or a reference at all, and can work directly with the channel outputs. There exists numerous algorithms of this kind. The most famous are the Sato algorithm [20] and the Godard algorithm [21]. The first basically works on a decision directed mode from the beginning of the adaptation process, whereas the second uses a modified cost function based solely on the amplitude of the channel outputs (no reference signal is specified). The interested reader on these types of algorithms can see [22].

  8. 8.

    QPSK is a digital constellation composed by four symbols, which can be represented as complex quantities: \(e^{j\pi /4}\), \(e^{j3\pi /4}\), \(e^{j5\pi /4}\) and \(e^{j7\pi /4}\).

  9. 9.

    If \(\mathbf B _{\mathbf x }\) is not diagonalizable, we can always find a Jordan decomposition [35] for it, and the result from Lemma 5. 2 is still valid [24].

  10. 10.

    We use \(\mathrm eig _i\left[\mathbf A \right]\) to denote the \(i\)-th eigenvalue of matrix \(\mathbf A \).

  11. 11.

    In (4.85) we used the fact that \(\mathbf A (n,j+1)\) and \(\tilde{\mathbf f }\left(\mathbf x (j)\right)\) are independent and that for two matrices \(\mathbf A \) and \(\mathbf B \) of appropriate dimensions, \(\mathrm tr \left[\mathbf A \mathbf B \right]=\mathrm tr \left[\mathbf B \mathbf A \right]\).

  12. 12.

    The fact that \(N(\varvec{ \mu })\) depends on \(\varvec{ \mu }\) is not relevant from the point of view of the stability, because it has no influence on the asymptotic behavior of \(\mathrm tr \left[\mathbf D (n,k+1)\right]\).

  13. 13.

    We will use the usual partial ordering defined for symmetric positive definite matrices [35].

  14. 14.

    It is in this place where we only keep the sufficiency and lose the necessity.

  15. 15.

    For Gaussian random variables we have the following result [39]:

    $$\begin{aligned} E[x_{1}x_2x_3x_4]=E[x_1x_2]E[x_3x_4]+E[x_1x_3]E[x_2x_4]+E[x_1x_4] E[x_2x_3]. \end{aligned}$$
  16. 16.

    For this, we need \(\mathbf R _\mathbf x \) to be strictly positive definite, which was assumed in Sect. 4.5.1.

  17. 17.

    Although in (4.130) we should put \(\approx \), in an abuse of notation we state it as an equality.

  18. 18.

    It should be emphasized that \(\Vert \mathbf e (n)\Vert ^2\) is not the same as the sum of the squares of the last \(K\) output estimation errors, \(\{e(i)\}_{i=n-K+1}^n\). Each component of the vector \(\mathbf e (n)\) is computed using the same filter estimate \(\mathbf w (n-1)\).

  19. 19.

    In Chap. 5 we will provide a deeper discussion about the properties of orthogonal projection operators.

  20. 20.

    Usual matrix inverting algorithms as Gaussian elimination require basically \(K^3\) multiplications [1]. The cost of APA is dominated by the \(K^2L\) multiplications and \(K^2(L-1)\) additions required to compute \(\mathbf X ^T(n) \mathbf X (n)\), since with \(L\gg K\) the cost of matrix inversion becomes less important.

  21. 21.

    Notice the similarity of this with a linear prediction problem from Sect.  2.5!!

References

  1. G.H. Golub, C.F. van Loan, Matrix Computations (The John Hopkins University Press, Baltimore, 1996)

    MATH  Google Scholar 

  2. W.W. Hager, Updating the inverse of a matrix. SIAM Review 31, 221–239 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  3. R. Nitzberg, Application of the normalized LMS algorithm to MSLC. IEEE Trans. Aerosp. Electron. Syst. AES-21, 79–91, (1985)

    Google Scholar 

  4. B. Widrow, S.D. Stearns, Adaptive Signal Processing (Prentice-Hall, Upper Saddle River, 1985)

    MATH  Google Scholar 

  5. A. Bhavani Sankar, D. Kumar, K. Seethalakshmi, Performance study of various adaptive filter algorithms for noise cancellation in respiratory signals. Signal Processing: An International Journal (SPIJ) 4, 267–278 (2010)

    Google Scholar 

  6. J. Glover Jr, Adaptive noise canceling applied to sinusoidal interferences. IEEE Trans. Acoust. Speech Signal Process. 25, 484–491 (1977)

    Article  Google Scholar 

  7. R. Quian Quiroga, Dataset #1: Human single-cell recording. http://www.vis.caltech.edu/ rodri/data.htm. (2003)

  8. C.S. Herrmann, T. Demiralp, Human EEG gamma oscillations in neuropsychiatric disorders. Clinical Neurophysiology 116, 2719–2733 (2005)

    Article  Google Scholar 

  9. R.D. Traub, M.A. Whittington, Cortical Oscillations in Health and Disease (Oxford University Press, New York, 2010)

    Book  Google Scholar 

  10. M.H. Costa, J.C. Moreira Bermudez, A noise resilient variable step-size LMS algorithm. Elsevier Signal Process. 88, 733–748 (2008)

    Article  MATH  Google Scholar 

  11. J.W. Kelly, J. L. Collinger, A. D. Degenhart, D. P. Siewiorek, A. Smailagic, W. Wang, Frequency tracking and variable bandwidth for line noise filtering without a reference. Proc. of IEEE EMBS, (Boston, 2011), pp. 7908–7911

    Google Scholar 

  12. A. Gersho, Adaptive filtering with binary reinforcement. IEEE Trans. Inform. Theory IT-30, 191–199 (1984)

    Google Scholar 

  13. W. A. Sethares and C. R. Johnson Jr., A comparison of two quantized state adaptive algorithms. IEEE Trans. Acoust. Speech Signal Process. ASSP-37, 138–143 (1989)

    Google Scholar 

  14. E. Eweda, Analysis and design of a signed regressor LMS algorithm for stationary and nonstationary adaptive filtering with correlated Gaussian data. IEEE Trans. Circuits Syst. 37, 1367–1374 (1990)

    Article  Google Scholar 

  15. J. Proakis, Digital Communications, 4th edn. (McGraw-Hill, New York, 2000)

    Google Scholar 

  16. S.U. Qureshi, Adaptive equalization. Proc. IEEE 73, 1349–1387 (1985)

    Article  Google Scholar 

  17. A.R. Bahai, B.R. Saltzberg, M. Ergen, Multi-carrier Digital Communications: Theory And Applications of OFDM, 2nd edn. (Springer, New York, 2004)

    Google Scholar 

  18. J. Liu, X. Lin, Equalization in high-speed communication systems. IEEE Circuits Syst. Mag. 4, 4–17 (2004)

    MathSciNet  Google Scholar 

  19. S. Haykin, Adaptive Filter Theory, 4th edn. (Prentice-Hall, Upper Saddle River, 2002)

    Google Scholar 

  20. Y. Sato, A method of self-recovering equalization for multi level amplitude modulation. IEEE Trans. Commun. 23, 679–682 (1975)

    Article  Google Scholar 

  21. D. Goddard, Self-recovering equalization and carrier tracking in two-dimensional data communication systems. IEEE Trans. Commun. 28, 1867–1875 (1980)

    Article  Google Scholar 

  22. R. Johnson, P. Schniter, T.J. Endres, J.D. Behm, D.R. Brown, R.A. Casas, Blind equalization using the constant modulus criterion: a review. Proc. IEEE 86, 1927–1950 (1998)

    Article  Google Scholar 

  23. P. Billingsley, Probability and Measure, 2nd edn. (Wiley-Interscience, New York, 1986)

    MATH  Google Scholar 

  24. T. Kailath, Linear Systems (Prentice-Hall, Englewood Cliffs, 1980)

    MATH  Google Scholar 

  25. L. Guo, Stability of recursive stochastic tracking algorithms. SIAM J. Control and Opt. 32, 1195–1225 (1994)

    Article  MATH  Google Scholar 

  26. J.A. Bucklew, T.G. Kurtz, W.A. Sethares, Weak convergence and local stability properties of fixed step size recursive algorithms. IEEE Trans. Inform. Theory 30, 966–978 (1993)

    Article  MathSciNet  Google Scholar 

  27. V. Solo, The stability of LMS. IEEE Trans. Signal Process. 45, 3017–3026 (1997)

    Article  Google Scholar 

  28. L. Guo, L. Ljung, G. Wang, Necessary and sufficient conditions for stability of LMS. IEEE Trans. Autom. Control 42, 761–770 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  29. L. Guo, L. Ljung, Exponential stability of general tracking algorithms. IEEE Trans. Autom. Control 40, 1376–1387 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  30. B. Widrow, J. Mc Cool, M.G. Larimore, C.R. Johnson, Stationary and nonstationary learning characteristic of the LMS adaptive filter. Proc. IEEE 64, 1151–1162 (1976)

    Article  MathSciNet  Google Scholar 

  31. A. Feuer and E. Weinstein, Convergence analysis of LMS filters with uncorrelated gaussian data. IEEE Trans. Acoust. Speech Signal Process. ASSP-33, 222–230 (1985).

    Google Scholar 

  32. J.E. Mazo, On the independence theory of equalizer convergence. Bell Syst. Techn. J. 58, 963–993 (1979)

    MathSciNet  MATH  Google Scholar 

  33. P.S.R. Diniz, Adaptive Filtering: Algorithms And Practical Implementation, 3rd edn. (Springer, Boston, 2008)

    MATH  Google Scholar 

  34. B. Farhang-Boroujeny, Adaptive Filters: Theory and Applications (John Wiley& Sons, New York, 1998)

    Google Scholar 

  35. R.A. Horn, C.R. Johnson, Matrix Analysis (Cambridge University Press, New York, 1990)

    MATH  Google Scholar 

  36. R. Price, A useful theorem for nonlinear devices having Gaussian inputs. IRE Trans. Inform. Theory IT-4, 69–72 (1958)

    Google Scholar 

  37. L. Rey Vega, H. Rey, J. Benesty, Stability analysis of a large family of adaptive filters. Elsevier Signal Process. 91, 2091–2100 (2011)

    Article  MATH  Google Scholar 

  38. T. Hu, A. Rosalsky, A. Volodin, On convergence properties of sums of dependent random variables under second moment and covariance restrictions. Stat. and Prob. Lett. 78, 1999–20 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  39. A. Papoulis, Probability, Random Variables, and Stochastic Processes (McGraw-Hill, New York, 1965)

    MATH  Google Scholar 

  40. T. Al-Naffouri, A. Sayed, Transient analysis of data normalized adaptive filters. IEEE Trans. Signal Process. 51, 639–652 (2003)

    Article  Google Scholar 

  41. M. Tarrab, A. Feuer, Convergence and performance analysis of the normalized LMS algorithm with uncorrelated gaussian data. IEEE Trans. Inform. Theory 34, 680–691 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  42. D.T. Slock, On the convergence behaviour of the LMS and the normalized LMS algorithms. IEEE Trans. Signal Process. 41, 2811–2825 (1993)

    Article  MATH  Google Scholar 

  43. M. Rupp, The behaviour of LMS and NLMS algorithms in the presence of spherically invariant processes. IEEE Trans. Signal Process. 41, 1149–1160 (1993)

    Article  MATH  Google Scholar 

  44. W. Sethares, I. Mareels, B. Anderson, C. Johnson, R. Bitmead, Excitation conditions for signed regressor least mean squares adaptation. IEEE Trans. Circuits Syst. 35, 613–624 (1988)

    Article  MathSciNet  Google Scholar 

  45. A.H. Sayed, Adaptive Filters (John Wiley& Sons, Hoboken, 2008)

    Book  Google Scholar 

  46. L. Rrtveit and J.H. Husy, A new prewhitening-based adaptive filter which converges to the Wiener-solution. Proc. Asilomar Conf. Sig. Syst. Comp. (Pacific Grove, 2009), pp. 1360–1364

    Google Scholar 

  47. C. Breining, P. Dreiscitel, E. Hansler, A. Mader, B. Nitsch, H. Puder, T. Schertler, G. Schmidt, J. Tilp, Acoustic echo control. An application of very-high-order adaptive filters. IEEE Signal Process. Mag. 16, 42–69 (1999)

    Google Scholar 

  48. N. Yousef, A. Sayed, A unified approach to the steady-state and tracking analyses of adaptive filters. IEEE Trans. Signal Process. 49, 314–324 (2001)

    Article  Google Scholar 

  49. K. Ozeki and T. Umeda, An adaptive filtering algorithm using an orthogonal projection to an affine subspace and its properties. Electron. Commun. in Japan 67-A, 19–27 (1984)

    Google Scholar 

  50. J. Apolinário Jr, M.L.R. Campos, P.S.R. Diniz, Convergence analysis of the binormalized data-reusing LMS algorithm. IEEE Trans. Signal Process. 48, 3235–3242 (2000)

    Article  Google Scholar 

  51. S.G. Sankaran, A.A.L. Beex, Convergence behavior of affine projection algorithms. IEEE Trans. Signal Process. 48, 1086–1096 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  52. S. Gay and S. Tavathia, The fast affine projection algorithm. Proc. IEEE ICASSP, (Detroit, 1995), pp. 3023–3026

    Google Scholar 

  53. H. Ding, Fast affine projection adaptation algorithms with stable and robust symmetric linear system solvers. IEEE Trans. Signal Process. 55, 1730–1740 (2007)

    Article  MathSciNet  Google Scholar 

  54. M. Tanaka, S. Makino, J. Kojima, A block exact fast affine projection algorithm. IEEE Trans. Speech Audio Process. 7, 79–86 (1999)

    Article  Google Scholar 

  55. M. Rupp, A.H. Sayed, A time-domain feedback analysis of filtered-error adaptive gradient algorithms. IEEE Trans. on Signal Process. 44, 1428–1439 (1996)

    Article  Google Scholar 

  56. H. Rey, L. Rey Vega, S. Tressens, J. Benesty, Variable explicit regularization in affine projection algorithm: robustness issues and optimal choice. IEEE Trans. Signal Process. 55, 2096–2109 (2007)

    Article  MathSciNet  Google Scholar 

  57. G. Meng, T. Elmedyb, S. Jensen, J. Jensen, Analysis of acoustic feedback/echo cancellation in multiple-microphone and single-loudspeaker systems using a power transfer function method. IEEE Trans. Signal Process. 59, 5774–5788 (2011)

    Article  MathSciNet  Google Scholar 

  58. M. Honig, M.K. Tsatsanis, Adaptive techniques for multiuser CDMA receivers. IEEE Signal Process. Mag. 17, 49–61 (2000)

    Article  Google Scholar 

  59. R.L. Calcavante, I. Yamada, Multiaccess interference suppression in orthogonal space-time block coded MIMO systems by adaptive projected subgradient method. IEEE Signal Process. Mag. 56, 1028–1042 (2007)

    Google Scholar 

  60. A. Zanella, M. Chiani, M. Win, Statistical analysis of steepest descend and LMS detection algorithms for MIMO systems. IEEE Trans. Veh. Technol. 60, 4667–4672 (2011)

    Article  Google Scholar 

  61. N.V. Thakor, Y.S. Zhu, Applications of adaptive filtering to ECG analysis: noise cancellation and arrhythmia detection. IEEE Trans. Biomed. Eng. 38, 785–794 (1991)

    Article  Google Scholar 

  62. S.M.M. Martens, M. Mischi, S.G. Oei, J.W.M. Bergmans, An improved adaptive power line interference canceller for electrocardiography. IEEE Trans. on Biomed. Eng. 53, 2220–2231 (2006)

    Article  Google Scholar 

  63. M. Bouchard, Multichannel affine and fast affine projection algorithms for active noise control and acoustic equalization systems. IEEE Trans. Speech Audio Process. 11, 54–60 (2003)

    Article  Google Scholar 

  64. E.P. Reddy, D.P. Das, K.M. Prabhu, Fast adaptive algorithms for active control of nonlinear noise processes. IEEE Trans. Signal Process. 56, 4530–4536 (2008)

    Article  MathSciNet  Google Scholar 

  65. J.S. Soo, K.K. Pang, New structures for adaptive filtering in subbands with critical sampling. IEEE Trans. Acoust. Speech Signal Process. 38, 373–376 (1990)

    Article  Google Scholar 

  66. M.R. Petraglia, R.G. Alves, P.S. Diniz, Multidelay block frequency domain adaptive filters. IEEE Trans. Signal Process. 48, 3316–3327 (2000)

    Article  MathSciNet  Google Scholar 

  67. S.S. Pradhan, V.U. Reddy, A new approach to subband adaptive filtering. IEEE Trans. Signal Process. 48, 655–664 (1999)

    Google Scholar 

  68. J. Benesty, C. Paleologu, S. Ciochina, On regularization in adaptive filtering. IEEE Trans. Audio, Speech, Lang. Process. 19, 1734–1742 (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Engineering, University of Buenos Aires, Paseo Colón 850, C1063ACV, Buenos Aires, Argentina

    Leonardo Rey Vega

  2. Department of Engineering, University of Leicester, University Road, Leicester, LE1 7RH, UK

    Hernan Rey

Authors
  1. Leonardo Rey Vega
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hernan Rey
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Leonardo Rey Vega .

Rights and permissions

Reprints and permissions

Copyright information

© 2013 The Author(s)

About this chapter

Cite this chapter

Rey Vega, L., Rey, H. (2013). Stochastic Gradient Adaptive Algorithms. In: A Rapid Introduction to Adaptive Filtering. SpringerBriefs in Electrical and Computer Engineering. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-30299-2_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-30299-2_4

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-30298-5

  • Online ISBN: 978-3-642-30299-2

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation