Adaptive Information Filtering with Error Entropy and Error Correntropy Criteria

  • Deniz Erdogmus
  • Weifeng Liu
Part of the Information Science and Statistics book series (ISS)


This chapter formulates a new cost function for adaptive filtering based on Renyi’s quadratic error entropy. The problem of estimating the linear system parameters \(\mathrm{\mathbf {w}} = {[{w}_{0},\ldots, {w}_{M-1}]}^{\mathrm{T}}\) in the setting of Figure 3.1 where x(n), and z(n) are random variables can be framed as model-based inference, because it relates measured data, uncertainty, and the functional description of the system and its parameters. The desired response z(n) can be thought of as being created by an unknown transformation of the input vector \(\mathrm{\mathbf {x}} = {[x(n),\ldots, x(n - M + 1)]}^{\mathrm{T}}\). Adaptive filtering theory [143, 284] addresses this problem using the MSE criterion applied to the error signal, \(e(n) = z(n) - f(\mathrm{\mathbf {w}},x(n))\)
$${J}_{w}(e(n)) = E[{(z(n) - f(\mathrm{\mathbf{w}},x(n)))}^{2}]$$
when the linear filter is a finite impulse response filter (FIR);
$$y(n) =\sum\limits_{k=0}^{M-1}{w}_{ k}x(n - k).$$


Cost Function Mean Square Error Probability Density Function Finite Impulse Response Performance Surface 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Aczél J., Daróczy Z., On measures of information and their characterizations, Mathematics in Science and Engineering, vol. 115, Academic Press, New York, 1975.Google Scholar
  2. 2.
    Ahmad I., Lin P., A nonparametric estimation of the entropy for absolutely continuous distributions, IEEE Trans. on Inf. Theor., 22:372–375, 1976.CrossRefMATHMathSciNetGoogle Scholar
  3. 4.
    Al-Naffouri T., Zerguine A., Bettayeb M., A unifying view of error nonlinearities in LMS adaptation, in Proc. ICASSP, vol. III, Seattle, pp. 1697–1700, May 1998.Google Scholar
  4. 6.
    Amari S., Nagoata H., Methods of information geometry, Mathematical Monographs, vol. 191, American Mathematical Society, Providence RI, 2000.Google Scholar
  5. 53.
    Chen B., Hu J., Pu L., Sun Z., Stochastic gradient algorithm under (h, ϕ)-entropy criterion, Circuits Syst. Signal Process., 26:941–960, 2007.CrossRefMATHMathSciNetGoogle Scholar
  6. 78.
    Douglas S., Meng H., Stochastic gradient adaptation under general error criteria, IEEE Trans. Signal Process., 42:1335–1351, 1994.CrossRefGoogle Scholar
  7. 84.
    Edmonson W., Srinivasan K., Wang C., Principe J. A global least square algorithm for adaptive IIR filtering, IEEE Trans. Circuits Syst., 45(3):379–384, 1996.CrossRefGoogle Scholar
  8. 87.
    Erdogmus D., Principe J.C., An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems, IEEE Trans. Signal Process., 50(7):1780–1786, 2002.CrossRefMathSciNetGoogle Scholar
  9. 88.
    Erdogmus D., J. Principe, Generalized information potential for adaptive systems training, IEEE Trans. Neural Netw., 13(5):1035–1044, 2002.CrossRefGoogle Scholar
  10. 104.
    Fox J., An R and S Companion to Applied Regression, Sage, London, 2002.Google Scholar
  11. 127.
    Hampel, F. R., Ronchetti E. M., Rousseau P. J., Stahel W. A., Robust Statistics: The Approach Based on Influence Functions. Wiley, New York, 1985.Google Scholar
  12. 133.
    Hardle W., Applied Nonparametric Regression, Econometric Society Monographs vol 19, Cambridge University Press, New York, 1990.Google Scholar
  13. 143.
    Haykin S., Adaptive Filter Theory, 4th Edition, Prentice Hall, Englewood Cliffs, NJ, 2002.Google Scholar
  14. 157.
    Huber, P.J., Robust Estimation of a Location Parameter. Ann. Math. Statist., 35:73–101, 1964.CrossRefMATHMathSciNetGoogle Scholar
  15. 165.
    Jenssen R., Erdogmus D., Hild II K., Principe J., Eltoft T., Information cut for clustering using a gradient descent approach, Pattern Recogn., 40:796–806, 2006.CrossRefGoogle Scholar
  16. 200.
    Liu W., Pokharel P., Principe J., Error entropy, correntropy and M-estimation, IEEE Int. Workshop on Machine Learning for Signal Processing, 2006.Google Scholar
  17. 201.
    Liu W., Pokharel P., Principe J., Correntropy: Properties and applications in non Gaussian signal processing, IEEE Trans. Sig. Proc., 55(11):5286–5298, 2007.CrossRefMathSciNetGoogle Scholar
  18. 220.
    Middleton D., Statistical-physical models of electromagnetic interference, IEEE Trans. Electromagn. Compat., EMC-19(3):106–126, Aug. 1977.CrossRefGoogle Scholar
  19. 223.
    Morejon R., An information theoretic approach to sonar automatic target recognition, Ph.D. dissertation, University of Florida, Spring 2003Google Scholar
  20. 244.
    Pei S., Tseng C., Least mean p-power error criterion for adaptive FIR filter, IEEE J. Selected Areas Commun., 12(9):1540–1547, 1994.CrossRefGoogle Scholar
  21. 275.
    Rubinstein R., Simulation and the Monte Carlo Method, John Wiley & Sons, New York, 1981.CrossRefMATHGoogle Scholar
  22. 284.
    Sayed A., Fundamentals of Adaptive Filters, John Wiley & Son, New York, 2003Google Scholar
  23. 297.
    Sidak Z., Sen P., Hajek J., Theory of Rank Tests, Academic Press, London, 1999.MATHGoogle Scholar
  24. 302.
    Singh A., Principe J., Using correntropy as a cost function in linear adaptive filters, Proc. IEEE IJCNN 2009, Atlanta, 2009.Google Scholar
  25. 310.
    Styblinski M., Tang T., Experiments in nonconvex optimization: Stochastic approximation with function smoothing and simulated annealing, Neural Netw., 3: 467–483, 1990.CrossRefGoogle Scholar
  26. 313.
    Tanrikuku O., Chambers J., Convergence and steady-state properties of the least-mean mixed norm (LMMN) adaptive algorithm, IEE Proc. -Vision, Image Signal Process., 143: 137–142, June 1996.CrossRefGoogle Scholar
  27. 328.
    Walach E., Widrow B., The least mean fourth (LMF) adaptive algorithm and its family, IEEE Trans. Inf. Theor., IT-30(2):275–283, 1984.CrossRefGoogle Scholar
  28. 332.
    Widrow B., S. Stearns, Adaptive Signal Processing, Prentice Hall, Englewood Cliffs, NJ, 1985.MATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Deniz Erdogmus
    • 1
  • Weifeng Liu
    • 1
  1. 1.Dept. Electrical Engineering & Biomedical EngineeringUniversity of FloridaGainesvilleUSA

Personalised recommendations