Advertisement

The Least-Mean-Square (LMS) Algorithm

  • Paulo Sergio Ramirez
Part of the The Kluwer International Series in Engineering and Computer Science book series (SECS, volume 694)

Abstract

The least-mean-square (LMS) is a search algorithm in which a simplification of the gradient vector computation is made possible by appropriately modifying the objective function [1]–[2]. The LMS algorithm, as well as others related to it, is widely used in various applications of adaptive filtering due to its computational simplicity [3]–[7]. The convergence characteristics of the LMS algorithm are examined in order to establish a range for the convergence factor that will guarantee stability. The convergence speed of the LMS is shown to be dependent on the eigenvalue spread of the input signal correlation matrix [2]–[6]. In this chapter, several properties of the LMS algorithm are discussed including the misadjustment in stationary and nonstationary environments [2]–[9] and tracking performance [10]–[12]. The analysis results are verified by a large number of simulation examples. Appendix A, section A.1, complements this chapter by analyzing the finite–wordlength effects in LMS algorithms.

Keywords

Gaussian White Noise Adaptive Filter Convergence Factor Unknown System Convergence Path 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    B. Widrow and M. E. Hoff, “Adaptive switching circuits,” WES COM Conv. Rec, pt. 4, pp. 96–140, 1960.Google Scholar
  2. 2.
    B. Widrow, J. M. McCool, M. G. Larimore, and C. R. Johnson Jr., “Stationary and nonstationary learning characteristics of the LMS adaptive filters,” Proceedings of the IEEE, vol. 64, pp. 1151–1162, Aug. 1976.MathSciNetCrossRefGoogle Scholar
  3. 3.
    G. Ungerboeck, “Theory on the speed of convergence in adaptive equalizers for digital communication,” IBM Journal on Research and Development, vol. 16, pp. 546–555, Nov. 1972.MATHCrossRefGoogle Scholar
  4. 4.
    J. E. Mazo, “On the independence theory of equalizer convergence,” The Bell System Technical Journal, vol. 58, pp. 963–993, May 1979.MathSciNetMATHGoogle Scholar
  5. 5.
    B. Widrow and S. D. Stearns, Adaptive Signal Processing, Prentice Hall, Englewood Cliffs, NJ, 1985.MATHGoogle Scholar
  6. 6.
    S. Haykin, Adaptive Filter Theory, Prentice Hall, Englewood Cliffs, NJ, 4th edition, 2002.Google Scholar
  7. 7.
    M. G. Bellanger, Adaptive Digital Filters and Signal Analysis, Marcel Dekker Inc., NY, 1987.Google Scholar
  8. 8.
    D. C. Farden, “Tracking properties of adaptive signal processing algorithms,” IEEE Trans, on Acoust., Speech, and Signal Processing, vol. ASSP-29, pp. 439–446, June 1981.MathSciNetCrossRefGoogle Scholar
  9. 9.
    B. Widrow and E. Walach, “On the statistical efficiency of the LMS algorithm with nonstationary inputs,” IEEE Trans, on Information Theory, vol. IT-30, pp. 211–221, March 1984.CrossRefGoogle Scholar
  10. 10.
    O. Macchi, “Optimization of adaptive identification for time varying filters,” IEEE Trans, on Automatic Control, vol. AC-31, pp. 283–287, March 1986.CrossRefGoogle Scholar
  11. 11.
    A. Benveniste, “Design of adaptive algorithms for the tracking of time varying systems,” Int. J. Adaptive Control and Signal Processing, vol. 1, pp. 3–29, Jan. 1987.MATHCrossRefGoogle Scholar
  12. 12.
    W. A. Gardner, “Nonstationary learning characteristics of the LMS algorithm,” IEEE Trans, on Circuits and Systems, vol. CAS-34, pp. 1199–1207, Oct. 1987.CrossRefGoogle Scholar
  13. 13.
    A. Papoulis, Probability, Random Variables, and Stochastic Processes, McGraw Hill, New York, NY, 3rd edition, 1991.Google Scholar
  14. 14.
    F. J. Gantmacher, The Theory of Matrices, Chelsea Publishing Company, vol. 2, New York, NY, 1964.Google Scholar
  15. 15.
    V. Solo, “The limiting behavior of LMS,” IEEE Trans, on Acoust., Speech, and Signal Processing, vol-37, pp. 1909–1922, Dec. 1989.MathSciNetMATHCrossRefGoogle Scholar
  16. 16.
    N. J. Bershad and O. M. Macchi, “Adaptive recovery of a chirped sinusoid in noise, Part 2: Performance of the LMS algorithm,” IEEE Trans, on Signal Processing, vol. 39, pp. 595–602, March 1991.CrossRefGoogle Scholar
  17. 17.
    D. G. Manolakis, V. K. Ingle, and S. M. Kogon, Statistical and Adaptive Signal Processing, McGraw Hill, New York, NY, 2000.Google Scholar
  18. 18.
    V. H. Nascimento and A. H. Sayed, “On the learning mechanism of adaptive filters,” IEEE Trans, on Signal Processing, vol. 48, pp. 1609–1625, June 2000.CrossRefGoogle Scholar
  19. 19.
    O. Macchi and E. Eweda, “Second–order convergence analysis of stochastic adaptive linear filter,” IEEE Trans, on Automatic Control, vol. AC-28, pp. 76–85, March 1983.MathSciNetCrossRefGoogle Scholar
  20. 20.
    S. Florian and A. Feuer, “Performance analysis of the LMS algorithm with a tapped delay line (two–dimensional case),” IEEE Trans, on Acoust., Speech, and Signal Processing, vol. ASSP-34, pp. 1542–1549, Dec. 1986.CrossRefGoogle Scholar
  21. 21.
    O. L. Frost III, “An algorithm for linearly constrained adaptive array processing,” Proceedings of the IEEE, vol. 60, pp. 926–935, Aug. 1972.CrossRefGoogle Scholar
  22. 22.
    J. A. Apolinario Jr., S. Werner, T. I. Laakso, and P. S. R. Diniz, “Constrained normalized adaptive filtering for CDMA mobile communications,” Proc. 1998 EUSIPCO–European Signal Processing Conference, Rhodes, Greece, pp. 2053–2056, Sept. 1998.Google Scholar
  23. 23.
    J. A. Apolinario Jr., M. L. R. de Campos, and C. P. Bernal Jr., “The Constrained conjugate–gradient algorithm,” IEEE Signal Processing Letters, vol. 7, pp.351–354, Dec. 2000.CrossRefGoogle Scholar
  24. 24.
    M. L. R. de Campos, S. Werner, and J. A. Apolinario Jr., “Constrained adaptation algorithms employing Householder transformations” IEEE Transactions on Signal Processing, accepted for publication 2002.Google Scholar
  25. 25.
    A. Feuer and E. Weinstein, “Convergence analysis of LMS filters with uncorrelated Gaussian data,” IEEE Trans, on Acoust., Speech, and Signal Processing, vol. ASSP-33, pp. 222–230, Feb. 1985.CrossRefGoogle Scholar
  26. 26.
    D. T. Slock, “On the convergence behavior of the LMS and normalized LMS algorithms,” IEEE Trans, on Signal Processing, vol. 40, pp. 2811–2825, Sept. 1993.CrossRefGoogle Scholar
  27. 27.
    W. A. Sethares, D. A. Lawrence, C. R. Johnson Jr., and R. R. Bitmead, “Parameter drift in LMS adaptive filters,” IEEE Trans, on Acoust., Speech, and Signal Processing, vol. ASSP-34, pp. 868–878, Aug. 1986.CrossRefGoogle Scholar
  28. 28.
    S. C. Douglas, “Exact expectation analysis of the LMS adaptive filter,” IEEE Trans, on Signal Processing, vol. 43, pp. 2863–2871, Dec. 1995.CrossRefGoogle Scholar
  29. 29.
    H. J. Butterweck, “Iterative analysis of the state–space weight fluctuations in LMS–type adaptive filters,” IEEE Trans, on Signal Processing, vol. 47, pp. 2558–2561, Sept. 1999.MATHCrossRefGoogle Scholar
  30. 30.
    B. Hassibi, A. H. Sayed, and T. Kailath, “H∞optimality of the LMS Algorithm,” IEEE Trans, on Signal Processing, vol. 44, pp. 267–280, Feb. 1996.CrossRefGoogle Scholar
  31. 31.
    O. J. Tobias, J. C. M. Bermudez, and N. J. Bershad, “Mean weight behavior of the filtered–X LMS algorithm,” IEEE Trans, on Signal Processing, vol. 48, pp. 1061–1075, April 2000.CrossRefGoogle Scholar
  32. 32.
    V. Solo, “The error variance of LMS with time varying weights,” IEEE Trans, on Signal Processing, vol. 40, pp. 803–813, April 1992.MATHCrossRefGoogle Scholar
  33. 33.
    S. U. Qureshi, “Adaptive Equalization,” Proceedings of the IEEE, vol-73, pp. 1349–1387, Sept. 1985.CrossRefGoogle Scholar
  34. 34.
    M. L. Honig, “Echo cancellation of voiceband data signals using recursive least squares and stochastic gradient algorithms,” IEEE Trans, on Communications, vol. COM-33, pp. 65–73, Jan. 1985.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2002

Authors and Affiliations

  • Paulo Sergio Ramirez
    • 1
  1. 1.Federal University of Rio de JaneiroRio de JaneiroBrazil

Personalised recommendations