The Least-Mean-Square (LMS) Algorithm

  • Paulo S. R. DinizEmail author


The least-mean-square (LMS) is a search algorithm in which simplification of the gradient vector computation is made possible by appropriately modifying the objective function [1, 2]. The review [3] explains the history behind the early proposal of the LMS algorithm, whereas [4] places into perspective the importance of this algorithm. The LMS algorithm, as well as others related to it, is widely used in various applications of adaptive filtering due to its computational simplicity [5, 6, 7, 8, 9]. The convergence characteristics of the LMS algorithm are examined in order to establish a range for the convergence factor that will guarantee stability.


  1. 1.
    B. Widrow, M.E. Hoff, Adaptive switching circuits. WESCOM Conv. Rec. 4, 96–140 (1960)Google Scholar
  2. 2.
    B. Widrow, J.M. McCool, M.G. Larimore, C.R. Johnson Jr., Stationary and nonstationary learning characteristics of the LMS adaptive filters. Proc. IEEE 64, 1151–1162 (1976)MathSciNetCrossRefGoogle Scholar
  3. 3.
    B. Widrow, D. Park, History of adaptive signal processing: Widrow’s group, in A Short History of Circuits and Systems, eds. by F. Maloberti, A.C. Davies (River Publishers, Delft, 2016)Google Scholar
  4. 4.
    P.S.R. Diniz, B. Widrow, History of adaptive filters, in A Short History of Circuits and Systems, eds. by F. Maloberti, A.C. Davies (River Publishers, Delft, 2016)Google Scholar
  5. 5.
    G. Ungerboeck, Theory on the speed of convergence in adaptive equalizers for digital communication. IBM J. Res. Dev. 16, 546–555 (1972)CrossRefGoogle Scholar
  6. 6.
    J.E. Mazo, On the independence theory of equalizer convergence. Bell Syst. Tech. J. 58, 963–993 (1979)MathSciNetCrossRefGoogle Scholar
  7. 7.
    B. Widrow, S.D. Stearns, Adaptive Signal Processing (Prentice Hall, Englewood Cliffs, 1985)zbMATHGoogle Scholar
  8. 8.
    S. Haykin, Adaptive Filter Theory, 4th edn. (Prentice Hall, Englewood Cliffs, 2002)zbMATHGoogle Scholar
  9. 9.
    M.G. Bellanger, Adaptive Digital Filters and Signal Analysis, 2nd edn. (Marcel Dekker Inc, New York, 2001)CrossRefGoogle Scholar
  10. 10.
    D.C. Farden, Racking properties of adaptive signal processing algorithms. IEEE Trans. Acoust. Speech Signal Process. (ASSP) 29, 439–446 (1981)Google Scholar
  11. 11.
    B. Widrow, E. Walach, On the statistical efficiency of the LMS algorithm with nonstationary inputs. IEEE Trans. Inform. Theor. (IT) 30, 211–221 (1984)Google Scholar
  12. 12.
    O. Macchi, Optimization of adaptive identification for time varying filters. IEEE Trans. Automat. Contr. (AC) 31, 283–287 (1986)Google Scholar
  13. 13.
    A. Benveniste, Design of adaptive algorithms for the tracking of time varying systems. Int. J. Adapt. Contr. Signal Process. 1, 3–29 (1987)CrossRefGoogle Scholar
  14. 14.
    W.A. Gardner, Nonstationary learning characteristics of the LMS algorithm. IEEE Trans. Circ. Syst. (CAS) 34, 1199–1207 (1987)Google Scholar
  15. 15.
    A. Papoulis, Probability, Random Variables, and Stochastic Processes, 3rd edn. (McGraw Hill, New York, 1991)zbMATHGoogle Scholar
  16. 16.
    F.J. Gantmacher, The Theory of Matrices, vol. 2 (Chelsea Publishing Company, New York, 1964)Google Scholar
  17. 17.
    G.H. Golub, C.F. Van Loan, Matrix Computations, 3rd edn. (John Hopkins University Press, Baltimore, 1996)zbMATHGoogle Scholar
  18. 18.
    V. Solo, The limiting behavior of LMS. IEEE Trans. Acoust. Speech Signal Process. 37, 1909–1922 (1989)MathSciNetCrossRefGoogle Scholar
  19. 19.
    N.J. Bershad, O.M. Macchi, Adaptive recovery of a chirped sinusoid in noise, part 2: performance of the LMS algorithm. IEEE Trans. Signal Process. 39, 595–602 (1991)CrossRefGoogle Scholar
  20. 20.
    D.H. Brandwood, A complex gradient operator and its application in adaptive array theory. IEE Proc. Parts F and G 130, 11–16 (1983)MathSciNetGoogle Scholar
  21. 21.
    A. Hjørungnes, D. Gesbert, Complex-valued matrix differentiation: techniques and key results. IEEE Trans. Signal Process. 55, 2740–2746 (2007)MathSciNetCrossRefGoogle Scholar
  22. 22.
    D.G. Manolakis, V.K. Ingle, S.M. Kogon, Statistical and Adaptive Signal Processing (McGraw Hill, New York, 2000)Google Scholar
  23. 23.
    O. Macchi, E. Eweda, Second-order convergence analysis of stochastic adaptive linear filter. IEEE Trans. Automat. Contr. (AC) 28, 76–85 (1983)Google Scholar
  24. 24.
    V.H. Nascimento, A.H. Sayed, On the learning mechanism of adaptive filters. IEEE Trans. Signal Process. 48, 1609–1625 (2000)CrossRefGoogle Scholar
  25. 25.
    S. Florian, A. Feuer, Performance analysis of the LMS algorithm with a tapped delay line (two-dimensional case). IEEE Trans. Acoust. Speech Signal Process. (ASSP) 34, 1542–1549 (1986)Google Scholar
  26. 26.
    O.L. Frost III, An algorithm for linearly constrained adaptive array processing. Proc. IEEE 60, 926–935 (1972)CrossRefGoogle Scholar
  27. 27.
    J.A. Apolinário Jr., S. Werner, T.I. Laakso, P.S.R. Diniz, Constrained normalized adaptive filtering for CDMA mobile communications, in Proceedings of 1998 EUSIPCO-European Signal Processing Conference, Rhodes, Greece (1998), pp. 2053–2056Google Scholar
  28. 28.
    J.A. Apolinário Jr., M.L.R. de Campos, C.P. Bernal O, The constrained conjugate-gradient algorithm. IEEE Signal Process. Lett. 7, 351–354 (2000)Google Scholar
  29. 29.
    M.L.R. de Campos, S. Werner, J.A. Apolinário Jr., Constrained adaptation algorithms employing Householder transformation. IEEE Trans. Signal Process. 50, 2187–2195 (2002)MathSciNetCrossRefGoogle Scholar
  30. 30.
    A. Feuer, E. Weinstein, Convergence analysis of LMS filters with uncorrelated Gaussian data. IEEE Trans. Acoust. Speech Signal Process. (ASSP) 33, 222–230 (1985)Google Scholar
  31. 31.
    D.T. Slock, On the convergence behavior of the LMS and normalized LMS algorithms. IEEE Trans. Signal Process. 40, 2811–2825 (1993)CrossRefGoogle Scholar
  32. 32.
    W.A. Sethares, D.A. Lawrence, C.R. Johnson Jr., R.R. Bitmead, Parameter drift in LMS adaptive filters. IEEE Trans. Acoust. Speech Signal Process. (ASSP) 34, 868–878 (1986)Google Scholar
  33. 33.
    S.C. Douglas, Exact expectation analysis of the LMS adaptive filter. IEEE Trans. Signal Process. 43, 2863–2871 (1995)CrossRefGoogle Scholar
  34. 34.
    H.J. Butterweck, Iterative analysis of the state-space weight fluctuations in LMS-type adaptive filters. IEEE Trans. Signal Process. 47, 2558–2561 (1999)CrossRefGoogle Scholar
  35. 35.
    B. Hassibi, A.H. Sayed, T. Kailath, \(H^{\infty }\) optimality of the LMS algorithm. IEEE Trans. Signal Process. 44, 267–280 (1996)CrossRefGoogle Scholar
  36. 36.
    O.J. Tobias, J.C.M. Bermudez, N.J. Bershad, Mean weight behavior of the filtered-X LMS algorithm. IEEE Trans. Signal Process. 48, 1061–1075 (2000)CrossRefGoogle Scholar
  37. 37.
    V. Solo, The error variance of LMS with time varying weights. IEEE Trans. Signal Process. 40, 803–813 (1992)CrossRefGoogle Scholar
  38. 38.
    S.U. Qureshi, Adaptive equalization. Proc. IEEE 73, 1349–1387 (1985)CrossRefGoogle Scholar
  39. 39.
    M.L. Honig, Echo cancellation of voiceband data signals using recursive least squares and stochastic gradient algorithms. IEEE Trans. Comm. (COM) 33, 65–73 (1985)Google Scholar
  40. 40.
    V.H. Nascimento, J.C.M. Bermudez, Probability of divergence for the least-mean fourth algorithm. IEEE Trans. Signal Proces. 54, 1376–1385 (2006)CrossRefGoogle Scholar
  41. 41.
    M.T.M. Silva, V.H. Nascimento, Improving tracking capability of adaptive filters via convex combination. IEEE Trans. Signal Proces. 56, 3137–3149 (2008)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Universidade Federal do Rio de JaneiroNiteróiBrazil

Personalised recommendations