Journal of Statistical Physics

, Volume 118, Issue 1–2, pp 145–176 | Cite as

Information and Entropy Flow in the Kalman–Bucy Filter

  • Sanjoy K. Mitter
  • Nigel J. Newton


We investigate the information theoretic properties of Kalman–Bucy filters in continuous time, developing notions of information supply, storage and dissipation. Introducing a concept of energy, we develop a physical analogy in which the unobserved signal describes a statistical mechanical system interacting with a heat bath. The abstract ‘universe’ comprising the signal and the heat bath obeys a non-increase law of entropy; however, with the introduction of partial observations, this law can be violated. The Kalman–Bucy filter behaves like a Maxwellian demon in this analogy, returning signal energy to the heat bath without causing entropy increase. This is made possible by the steady supply of new information. In a second analogy the signal and filter interact, setting up a stationary non-equilibrium state, in which energy flows between the heat bath, the signal and the filter without causing any overall entropy increase. We introduce a rate of interactive entropy flow that isolates the statistical mechanics of this flow from marginal effects. Both analogies provide quantitative examples of Landauer’s Principle.


Information theory Landauer’s principle non-equilibrium statistical mechanics statistical filtering 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Bennett, CH. 1982The thermodynamics of computation—a review. IntJ. Theoret. Phys.21905940Google Scholar
  2. Bertini, L., DeSole, A., Gabrielli, D., Jona-Lasinio, G., Landim, C. 2002Macroscopic fluctuation theory for stationary non-equilibrium states. J. StatPhy.107635675Google Scholar
  3. Borkar VS., Chari RT., Mitter SK. (1988). Stochastic quantization of field theory in finite and infinite volume. J. Func. Anal. 81Google Scholar
  4. R.W. Brockett and J.C.Willems, Stochastic control and the second law of thermodynamics, Proceedings of the 17th IEEE Conference on Decision and Control (San Diego CA, IEEE, 1979), pp.1007–1011Google Scholar
  5. Davis, M.H.A. 1977Linear Estimation and Stochastic ControlChapman and HallLondonGoogle Scholar
  6. Duncan, TE. 1970On the calculation of mutual informationSIAM J. Appl. Math.19215220Google Scholar
  7. Feng, X., Loparo, KA., Fang, Y. 1997Optimal state estimation for stochastic systems: and information theoretic approachIEEE Trans.Automatic Control.42771785Google Scholar
  8. Gallavotti, G. 1999Statistical Mechanics—A Short TreatiseSpringer-VerlagBerlin Heidelberg/NewyorkGoogle Scholar
  9. Gaveau, B., Schulman, LS. 2003Creation, dissipation and recycling of resources in non-equilibrium systemsJ. Stat. Phys.11013171367Google Scholar
  10. Jazwinski, A.H. 1970Stochastic Processes and Filtering TheoryAcademic PressLondonGoogle Scholar
  11. Johnson, JB. 1928Thermal agitation of electricity in conductorsPhys.Rev.3297109Google Scholar
  12. Jona-Lasinio, G., Mitter, PK. 1985On the stochastic quantization of field theoryComm. Math. Phys.101409436Google Scholar
  13. Kalata, P., Priemer, R. 1979Linear prediction, filtering and smoothing: an information theoretic approachInf.Sci.17114Google Scholar
  14. Karatzas, I., Shreve, S. 1991Brownian Motion and Stochastic CalculusSpringer-VerlagBerlin/Heidelberg/New YorkGoogle Scholar
  15. Kurtz, TG., Ocone, DL. 1988Unique characterization of conditional distributions in non-linear filteringAnn.Prob.1680107Google Scholar
  16. Landauer, R. 1961Dissipation and heat generation in the computing processIBM J. Res. Dev.5183191Google Scholar
  17. Lebowitz, L., Spohn, H., Gallavotti, A. 1999Cohen type symmetry in the large deviation functional for stochastic dynamics. J. StatPhys.95333366Google Scholar
  18. Liptser, RS., Shiryayev, AN. 1977Statistics of Random Processes 1—General TheorySpringer-VerlagBerlin/Heidelberg/New YorkGoogle Scholar
  19. Liptser, R.S., Shiryayev, A.N. 1978Statistics of Random Processes 2—ApplicationsSpringer-VerlagBerlin/Heidelberg/New YorkGoogle Scholar
  20. Maxwell, J.C. 1871Theory of HeatLongmansLondonGoogle Scholar
  21. Mitter SK., Newton NJ. (2003). A Variational approach to non-linear estimation. SIAM J. Control Optim. 42:1813–1833Google Scholar
  22. Mitter, S.K., Vinter, R.B. 1975Filtering for linear stochastic hereditary differential systems, Lecture Notes in Economics and Mathematical Systems, 107 Control Theory, Numerical Methods and Computer ModellingSpringer-VerlagBerlin/Heidelberg/New York121Google Scholar
  23. Nyquist, H. 1928Thermal agitation of electric charge in conductorsPhys.Rev.32110113Google Scholar
  24. Propp, M.B. 1985The Thermodynamic Properties of Markov Processes, Ph.D.ThesisMassachusetts Institute of TechnologyCambridge, MA, USAGoogle Scholar
  25. Ruelle, D. 1996Posititivity of entropy production in non-equilibrium statistical mechanicsJ.Stat.Phys.85123Google Scholar
  26. E.Mayer-Wolf and M.Zakai, On a formula relating the Shannon information to the Fisher information for the filtering problem, in Filtering and Control of Random Processes, H. Korezlioglu, G. Mazziotto, and S. Szpirglas, eds., Lecture Notes in Control and Information Sciences 61 (Springer, 1984), pp.164–171Google Scholar
  27. Tomita, Y., Omatu, S., Soeda, T. 1976An application of the information theory to filtering problemInf.Sci.111327Google Scholar
  28. Willems, JC. 1972Dissipative dynamical systems, part i: general theoryArch. Ratinal. Mech.Anal.45321351Google Scholar

Copyright information

© Springer Science+Business Media, Inc. 2005

Authors and Affiliations

  1. 1.Department of Electrical Engineering and Computer Science, and Laboratory for Information and Decision SystemsMassachusetts Institute of TechnologyCambridgeUnited States of America
  2. 2.Department of Electronic Systems EngineeringUniversity of EssexColchesterUnited Kingdom

Personalised recommendations