Stochastic Gradient Adaptive Algorithms

  • Leonardo Rey Vega
  • Hernan Rey
Part of the SpringerBriefs in Electrical and Computer Engineering book series (BRIEFSELECTRIC)


One way to construct adaptive algorithms leads to the so called Stochastic Gradient algorithms which will be the subject of this chapter. The most important algorithm in this family, the Least Mean Square algorithm (LMS), is obtained from the SD algorithm, employing suitable estimators of the correlation matrix and cross correlation vector. Other important algorithms as the Normalized Least Mean Square (NLMS) or the Affine Projection (APA) algorithms are obtained from straightforward generalizations of the LMS algorithm. One of the most useful properties of adaptive algorithms is the ability of tracking variations in the signals statistics. As they are implemented using stochastic signals, the update directions in these adaptive algorithms become subject to random fluctuations called gradient noise. This will lead to the question regarding the performance (in statistical terms) of these systems. In this chapter we will try to give a succinct introduction to this kind of adaptive filter and to its more relevant characteristics.


Adaptive Algorithm Little Mean Square Adaptive Filter Mean Square Deviation Steady State Error 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    G.H. Golub, C.F. van Loan, Matrix Computations (The John Hopkins University Press, Baltimore, 1996)zbMATHGoogle Scholar
  2. 2.
    W.W. Hager, Updating the inverse of a matrix. SIAM Review 31, 221–239 (1989)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    R. Nitzberg, Application of the normalized LMS algorithm to MSLC. IEEE Trans. Aerosp. Electron. Syst. AES-21, 79–91, (1985)Google Scholar
  4. 4.
    B. Widrow, S.D. Stearns, Adaptive Signal Processing (Prentice-Hall, Upper Saddle River, 1985)zbMATHGoogle Scholar
  5. 5.
    A. Bhavani Sankar, D. Kumar, K. Seethalakshmi, Performance study of various adaptive filter algorithms for noise cancellation in respiratory signals. Signal Processing: An International Journal (SPIJ) 4, 267–278 (2010)Google Scholar
  6. 6.
    J. Glover Jr, Adaptive noise canceling applied to sinusoidal interferences. IEEE Trans. Acoust. Speech Signal Process. 25, 484–491 (1977)CrossRefGoogle Scholar
  7. 7.
    R. Quian Quiroga, Dataset #1: Human single-cell recording. rodri/data.htm. (2003)
  8. 8.
    C.S. Herrmann, T. Demiralp, Human EEG gamma oscillations in neuropsychiatric disorders. Clinical Neurophysiology 116, 2719–2733 (2005)CrossRefGoogle Scholar
  9. 9.
    R.D. Traub, M.A. Whittington, Cortical Oscillations in Health and Disease (Oxford University Press, New York, 2010)CrossRefGoogle Scholar
  10. 10.
    M.H. Costa, J.C. Moreira Bermudez, A noise resilient variable step-size LMS algorithm. Elsevier Signal Process. 88, 733–748 (2008)zbMATHCrossRefGoogle Scholar
  11. 11.
    J.W. Kelly, J. L. Collinger, A. D. Degenhart, D. P. Siewiorek, A. Smailagic, W. Wang, Frequency tracking and variable bandwidth for line noise filtering without a reference. Proc. of IEEE EMBS, (Boston, 2011), pp. 7908–7911Google Scholar
  12. 12.
    A. Gersho, Adaptive filtering with binary reinforcement. IEEE Trans. Inform. Theory IT-30, 191–199 (1984)Google Scholar
  13. 13.
    W. A. Sethares and C. R. Johnson Jr., A comparison of two quantized state adaptive algorithms. IEEE Trans. Acoust. Speech Signal Process. ASSP-37, 138–143 (1989)Google Scholar
  14. 14.
    E. Eweda, Analysis and design of a signed regressor LMS algorithm for stationary and nonstationary adaptive filtering with correlated Gaussian data. IEEE Trans. Circuits Syst. 37, 1367–1374 (1990)CrossRefGoogle Scholar
  15. 15.
    J. Proakis, Digital Communications, 4th edn. (McGraw-Hill, New York, 2000)Google Scholar
  16. 16.
    S.U. Qureshi, Adaptive equalization. Proc. IEEE 73, 1349–1387 (1985)CrossRefGoogle Scholar
  17. 17.
    A.R. Bahai, B.R. Saltzberg, M. Ergen, Multi-carrier Digital Communications: Theory And Applications of OFDM, 2nd edn. (Springer, New York, 2004)Google Scholar
  18. 18.
    J. Liu, X. Lin, Equalization in high-speed communication systems. IEEE Circuits Syst. Mag. 4, 4–17 (2004)MathSciNetGoogle Scholar
  19. 19.
    S. Haykin, Adaptive Filter Theory, 4th edn. (Prentice-Hall, Upper Saddle River, 2002)Google Scholar
  20. 20.
    Y. Sato, A method of self-recovering equalization for multi level amplitude modulation. IEEE Trans. Commun. 23, 679–682 (1975)CrossRefGoogle Scholar
  21. 21.
    D. Goddard, Self-recovering equalization and carrier tracking in two-dimensional data communication systems. IEEE Trans. Commun. 28, 1867–1875 (1980)CrossRefGoogle Scholar
  22. 22.
    R. Johnson, P. Schniter, T.J. Endres, J.D. Behm, D.R. Brown, R.A. Casas, Blind equalization using the constant modulus criterion: a review. Proc. IEEE 86, 1927–1950 (1998)CrossRefGoogle Scholar
  23. 23.
    P. Billingsley, Probability and Measure, 2nd edn. (Wiley-Interscience, New York, 1986)zbMATHGoogle Scholar
  24. 24.
    T. Kailath, Linear Systems (Prentice-Hall, Englewood Cliffs, 1980)zbMATHGoogle Scholar
  25. 25.
    L. Guo, Stability of recursive stochastic tracking algorithms. SIAM J. Control and Opt. 32, 1195–1225 (1994)zbMATHCrossRefGoogle Scholar
  26. 26.
    J.A. Bucklew, T.G. Kurtz, W.A. Sethares, Weak convergence and local stability properties of fixed step size recursive algorithms. IEEE Trans. Inform. Theory 30, 966–978 (1993)MathSciNetCrossRefGoogle Scholar
  27. 27.
    V. Solo, The stability of LMS. IEEE Trans. Signal Process. 45, 3017–3026 (1997)CrossRefGoogle Scholar
  28. 28.
    L. Guo, L. Ljung, G. Wang, Necessary and sufficient conditions for stability of LMS. IEEE Trans. Autom. Control 42, 761–770 (1997)MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    L. Guo, L. Ljung, Exponential stability of general tracking algorithms. IEEE Trans. Autom. Control 40, 1376–1387 (1995)MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    B. Widrow, J. Mc Cool, M.G. Larimore, C.R. Johnson, Stationary and nonstationary learning characteristic of the LMS adaptive filter. Proc. IEEE 64, 1151–1162 (1976)MathSciNetCrossRefGoogle Scholar
  31. 31.
    A. Feuer and E. Weinstein, Convergence analysis of LMS filters with uncorrelated gaussian data. IEEE Trans. Acoust. Speech Signal Process. ASSP-33, 222–230 (1985).Google Scholar
  32. 32.
    J.E. Mazo, On the independence theory of equalizer convergence. Bell Syst. Techn. J. 58, 963–993 (1979)MathSciNetzbMATHGoogle Scholar
  33. 33.
    P.S.R. Diniz, Adaptive Filtering: Algorithms And Practical Implementation, 3rd edn. (Springer, Boston, 2008)zbMATHGoogle Scholar
  34. 34.
    B. Farhang-Boroujeny, Adaptive Filters: Theory and Applications (John Wiley& Sons, New York, 1998)Google Scholar
  35. 35.
    R.A. Horn, C.R. Johnson, Matrix Analysis (Cambridge University Press, New York, 1990)zbMATHGoogle Scholar
  36. 36.
    R. Price, A useful theorem for nonlinear devices having Gaussian inputs. IRE Trans. Inform. Theory IT-4, 69–72 (1958)Google Scholar
  37. 37.
    L. Rey Vega, H. Rey, J. Benesty, Stability analysis of a large family of adaptive filters. Elsevier Signal Process. 91, 2091–2100 (2011)zbMATHCrossRefGoogle Scholar
  38. 38.
    T. Hu, A. Rosalsky, A. Volodin, On convergence properties of sums of dependent random variables under second moment and covariance restrictions. Stat. and Prob. Lett. 78, 1999–20 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  39. 39.
    A. Papoulis, Probability, Random Variables, and Stochastic Processes (McGraw-Hill, New York, 1965)zbMATHGoogle Scholar
  40. 40.
    T. Al-Naffouri, A. Sayed, Transient analysis of data normalized adaptive filters. IEEE Trans. Signal Process. 51, 639–652 (2003)CrossRefGoogle Scholar
  41. 41.
    M. Tarrab, A. Feuer, Convergence and performance analysis of the normalized LMS algorithm with uncorrelated gaussian data. IEEE Trans. Inform. Theory 34, 680–691 (1988)MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    D.T. Slock, On the convergence behaviour of the LMS and the normalized LMS algorithms. IEEE Trans. Signal Process. 41, 2811–2825 (1993)zbMATHCrossRefGoogle Scholar
  43. 43.
    M. Rupp, The behaviour of LMS and NLMS algorithms in the presence of spherically invariant processes. IEEE Trans. Signal Process. 41, 1149–1160 (1993)zbMATHCrossRefGoogle Scholar
  44. 44.
    W. Sethares, I. Mareels, B. Anderson, C. Johnson, R. Bitmead, Excitation conditions for signed regressor least mean squares adaptation. IEEE Trans. Circuits Syst. 35, 613–624 (1988)MathSciNetCrossRefGoogle Scholar
  45. 45.
    A.H. Sayed, Adaptive Filters (John Wiley& Sons, Hoboken, 2008)CrossRefGoogle Scholar
  46. 46.
    L. Rrtveit and J.H. Husy, A new prewhitening-based adaptive filter which converges to the Wiener-solution. Proc. Asilomar Conf. Sig. Syst. Comp. (Pacific Grove, 2009), pp. 1360–1364Google Scholar
  47. 47.
    C. Breining, P. Dreiscitel, E. Hansler, A. Mader, B. Nitsch, H. Puder, T. Schertler, G. Schmidt, J. Tilp, Acoustic echo control. An application of very-high-order adaptive filters. IEEE Signal Process. Mag. 16, 42–69 (1999)Google Scholar
  48. 48.
    N. Yousef, A. Sayed, A unified approach to the steady-state and tracking analyses of adaptive filters. IEEE Trans. Signal Process. 49, 314–324 (2001)CrossRefGoogle Scholar
  49. 49.
    K. Ozeki and T. Umeda, An adaptive filtering algorithm using an orthogonal projection to an affine subspace and its properties. Electron. Commun. in Japan 67-A, 19–27 (1984)Google Scholar
  50. 50.
    J. Apolinário Jr, M.L.R. Campos, P.S.R. Diniz, Convergence analysis of the binormalized data-reusing LMS algorithm. IEEE Trans. Signal Process. 48, 3235–3242 (2000)CrossRefGoogle Scholar
  51. 51.
    S.G. Sankaran, A.A.L. Beex, Convergence behavior of affine projection algorithms. IEEE Trans. Signal Process. 48, 1086–1096 (2000)MathSciNetzbMATHCrossRefGoogle Scholar
  52. 52.
    S. Gay and S. Tavathia, The fast affine projection algorithm. Proc. IEEE ICASSP, (Detroit, 1995), pp. 3023–3026Google Scholar
  53. 53.
    H. Ding, Fast affine projection adaptation algorithms with stable and robust symmetric linear system solvers. IEEE Trans. Signal Process. 55, 1730–1740 (2007)MathSciNetCrossRefGoogle Scholar
  54. 54.
    M. Tanaka, S. Makino, J. Kojima, A block exact fast affine projection algorithm. IEEE Trans. Speech Audio Process. 7, 79–86 (1999)CrossRefGoogle Scholar
  55. 55.
    M. Rupp, A.H. Sayed, A time-domain feedback analysis of filtered-error adaptive gradient algorithms. IEEE Trans. on Signal Process. 44, 1428–1439 (1996)CrossRefGoogle Scholar
  56. 56.
    H. Rey, L. Rey Vega, S. Tressens, J. Benesty, Variable explicit regularization in affine projection algorithm: robustness issues and optimal choice. IEEE Trans. Signal Process. 55, 2096–2109 (2007)MathSciNetCrossRefGoogle Scholar
  57. 57.
    G. Meng, T. Elmedyb, S. Jensen, J. Jensen, Analysis of acoustic feedback/echo cancellation in multiple-microphone and single-loudspeaker systems using a power transfer function method. IEEE Trans. Signal Process. 59, 5774–5788 (2011)MathSciNetCrossRefGoogle Scholar
  58. 58.
    M. Honig, M.K. Tsatsanis, Adaptive techniques for multiuser CDMA receivers. IEEE Signal Process. Mag. 17, 49–61 (2000)CrossRefGoogle Scholar
  59. 59.
    R.L. Calcavante, I. Yamada, Multiaccess interference suppression in orthogonal space-time block coded MIMO systems by adaptive projected subgradient method. IEEE Signal Process. Mag. 56, 1028–1042 (2007)Google Scholar
  60. 60.
    A. Zanella, M. Chiani, M. Win, Statistical analysis of steepest descend and LMS detection algorithms for MIMO systems. IEEE Trans. Veh. Technol. 60, 4667–4672 (2011)CrossRefGoogle Scholar
  61. 61.
    N.V. Thakor, Y.S. Zhu, Applications of adaptive filtering to ECG analysis: noise cancellation and arrhythmia detection. IEEE Trans. Biomed. Eng. 38, 785–794 (1991)CrossRefGoogle Scholar
  62. 62.
    S.M.M. Martens, M. Mischi, S.G. Oei, J.W.M. Bergmans, An improved adaptive power line interference canceller for electrocardiography. IEEE Trans. on Biomed. Eng. 53, 2220–2231 (2006)CrossRefGoogle Scholar
  63. 63.
    M. Bouchard, Multichannel affine and fast affine projection algorithms for active noise control and acoustic equalization systems. IEEE Trans. Speech Audio Process. 11, 54–60 (2003)CrossRefGoogle Scholar
  64. 64.
    E.P. Reddy, D.P. Das, K.M. Prabhu, Fast adaptive algorithms for active control of nonlinear noise processes. IEEE Trans. Signal Process. 56, 4530–4536 (2008)MathSciNetCrossRefGoogle Scholar
  65. 65.
    J.S. Soo, K.K. Pang, New structures for adaptive filtering in subbands with critical sampling. IEEE Trans. Acoust. Speech Signal Process. 38, 373–376 (1990)CrossRefGoogle Scholar
  66. 66.
    M.R. Petraglia, R.G. Alves, P.S. Diniz, Multidelay block frequency domain adaptive filters. IEEE Trans. Signal Process. 48, 3316–3327 (2000)MathSciNetCrossRefGoogle Scholar
  67. 67.
    S.S. Pradhan, V.U. Reddy, A new approach to subband adaptive filtering. IEEE Trans. Signal Process. 48, 655–664 (1999)Google Scholar
  68. 68.
    J. Benesty, C. Paleologu, S. Ciochina, On regularization in adaptive filtering. IEEE Trans. Audio, Speech, Lang. Process. 19, 1734–1742 (2011)Google Scholar

Copyright information

© The Author(s) 2013

Authors and Affiliations

  1. 1.School of EngineeringUniversity of Buenos AiresBuenos AiresArgentina
  2. 2.Department of EngineeringUniversity of LeicesterLeicesterUK

Personalised recommendations