Algorithms for Entropy and Correntropy Adaptation with Applications to Linear Systems

  • Deniz Erdogmus
  • Seungju Han
  • Abhishek Singh
Part of the Information Science and Statistics book series (ISS)


This chapter develops several batch and online learning algorithms for the error entropy criterion (EEC) that are counterparts to the most widely used algorithms for the mean square error criterion (MSE). Because the chapter assumes knowledge of adaptive filter design, readers unfamiliar with this topic should seek a textbook such as [332] or [253] for a review of fundamentals. But the treatment does not require an in-depth knowledge of this field. The case studies in this chapter address only adaptation of linear systems, not because entropic costs are particularly useful for the linear model, but because the solutions for linear systems are well understood and performance comparisons can be easily drawn. This chapter also considers applications of fast evaluations of the IP using the fast Gauss transform and incomplete Cholesky decomposition, and ends with an application of the error correntropy criterion (ECC) to adaptive noise cancellation.


Mean Square Error Finite Impulse Response Adaptive Filter Recursive Less Square Kernel Size 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Aczél J., Daróczy Z., On measures of information and their characterizations, Mathematics in Science and Engineering, vol. 115, Academic Press, New York, 1975.Google Scholar
  2. 2.
    Ahmad I., Lin P., A nonparametric estimation of the entropy for absolutely continuous distributions, IEEE Trans. on Inf. Theor., 22:372–375, 1976.CrossRefMATHMathSciNetGoogle Scholar
  3. 88.
    Erdogmus D., J. Principe, Generalized information potential for adaptive systems training, IEEE Trans. Neural Netw., 13(5):1035–1044, 2002.CrossRefGoogle Scholar
  4. 90.
    Erdogmus D., Principe J., Kim S., Sanchez J., A recursive Renyi’s entropy estimator, Proc. IEEE Workshop on Neural Networks for Signal Process., Martigni, Switzerland, pp. 209–217, 2002.Google Scholar
  5. 116.
    Golub G., Van Loan C., Matrix Computation, 3rd ed. The Johns Hopkins University Press, Baltimore, Maryland, 1996.Google Scholar
  6. 118.
    Granas A., Dugundji J., Fixed Point Theory, Springer-Verlag, New York, 2003.CrossRefMATHGoogle Scholar
  7. 128.
    Han S., Rao S., Erdogmus D., Principe J., A minimum error entropy algorithm with self adjusting stepsize (MEE-SAS), Signal Process. 87:2733–2745.Google Scholar
  8. 129.
    Han S., Principe J., A Fixed-point Minimum Error Entropy Algorithm, in Proc. IEEE Int. Workshop on Machine Learning for Signal Processing, Maynooth, Ireland, 2006.Google Scholar
  9. 130.
    Han S., Rao S., Jeong K., Principe J., A normalized minimum error entropy stochastic algorithm”, Proc. IEEE Int. Conf. Acoustics Speech and Signal Processing, Toulouse, 2006.Google Scholar
  10. 131.
    Han S., A Family of Minimum Renyi’s Error Entropy Algorithm for Information Processing, Ph.D. dissertation, University of Florida, Summer, 2007.Google Scholar
  11. 143.
    Haykin S., Adaptive Filter Theory, 4th Edition, Prentice Hall, Englewood Cliffs, NJ, 2002.Google Scholar
  12. 192.
    Kushner H., Yin G., Stochastic approximation and recursive algorithms and applications, Application of Mathematics series, vol. 35, Springer, New York, 2003.Google Scholar
  13. 206.
    Lyapunov A. Stability of motion, Academic Press, New York, 1966MATHGoogle Scholar
  14. 253.
    Principe J., Euliano N., Lefebvre C., Neural Systems: Fundamentals through Simulations, CD-ROM textbook, John Wiley, New York, 2000.Google Scholar
  15. 258.
    Rao Y., D. Erdogmus, G.Y. Rao, J.C. Principe, Fast error whitening algorithms for system identification and control with noisy data, Neurocomputing, 69:158–181, 2006.CrossRefGoogle Scholar
  16. 302.
    Singh A., Principe J., Using correntropy as a cost function in linear adaptive filters, Proc. IEEE IJCNN 2009, Atlanta, 2009.Google Scholar
  17. 332.
    Widrow B., S. Stearns, Adaptive Signal Processing, Prentice Hall, Englewood Cliffs, NJ, 1985.MATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Deniz Erdogmus
    • 1
  • Seungju Han
    • 1
  • Abhishek Singh
    • 1
  1. 1.Dept. Electrical Engineering & Biomedical EngineeringUniversity of FloridaGainesvilleUSA

Personalised recommendations