Abstract
We investigate the information theoretic properties of Kalman–Bucy filters in continuous time, developing notions of information supply, storage and dissipation. Introducing a concept of energy, we develop a physical analogy in which the unobserved signal describes a statistical mechanical system interacting with a heat bath. The abstract ‘universe’ comprising the signal and the heat bath obeys a non-increase law of entropy; however, with the introduction of partial observations, this law can be violated. The Kalman–Bucy filter behaves like a Maxwellian demon in this analogy, returning signal energy to the heat bath without causing entropy increase. This is made possible by the steady supply of new information. In a second analogy the signal and filter interact, setting up a stationary non-equilibrium state, in which energy flows between the heat bath, the signal and the filter without causing any overall entropy increase. We introduce a rate of interactive entropy flow that isolates the statistical mechanics of this flow from marginal effects. Both analogies provide quantitative examples of Landauer’s Principle.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
CH. Bennett (1982) ArticleTitleThe thermodynamics of computation—a review. Int J. Theoret. Phys. 21 905–940
L. Bertini A. DeSole D. Gabrielli G. Jona-Lasinio C. Landim (2002) ArticleTitleMacroscopic fluctuation theory for stationary non-equilibrium states. J. Stat Phy. 107 635–675
Borkar VS., Chari RT., Mitter SK. (1988). Stochastic quantization of field theory in finite and infinite volume. J. Func. Anal. 81
R.W. Brockett and J.C.Willems, Stochastic control and the second law of thermodynamics, Proceedings of the 17th IEEE Conference on Decision and Control (San Diego CA, IEEE, 1979), pp.1007–1011
M.H.A. Davis (1977) Linear Estimation and Stochastic Control Chapman and Hall London
TE. Duncan (1970) ArticleTitleOn the calculation of mutual information SIAM J. Appl. Math. 19 215–220
X. Feng KA. Loparo Y. Fang (1997) ArticleTitleOptimal state estimation for stochastic systems: and information theoretic approach IEEE Trans.Automatic Control. 42 771–785
G. Gallavotti (1999) Statistical Mechanics—A Short Treatise Springer-Verlag Berlin Heidelberg/Newyork
B. Gaveau LS. Schulman (2003) ArticleTitleCreation, dissipation and recycling of resources in non-equilibrium systems J. Stat. Phys. 110 1317–1367
A.H. Jazwinski (1970) Stochastic Processes and Filtering Theory Academic Press London
JB. Johnson (1928) ArticleTitleThermal agitation of electricity in conductors Phys.Rev. 32 97–109
G. Jona-Lasinio PK. Mitter (1985) ArticleTitleOn the stochastic quantization of field theory Comm. Math. Phys. 101 409–436
P. Kalata R. Priemer (1979) ArticleTitleLinear prediction, filtering and smoothing: an information theoretic approach Inf.Sci. 17 1–14
I. Karatzas S. Shreve (1991) Brownian Motion and Stochastic Calculus Springer-Verlag Berlin/Heidelberg/New York
TG. Kurtz DL. Ocone (1988) ArticleTitleUnique characterization of conditional distributions in non-linear filtering Ann.Prob. 16 80–107
R. Landauer (1961) ArticleTitleDissipation and heat generation in the computing process IBM J. Res. Dev. 5 183–191
L. Lebowitz H. Spohn A. Gallavotti (1999) ArticleTitleCohen type symmetry in the large deviation functional for stochastic dynamics. J. Stat Phys. 95 333–366
RS. Liptser AN. Shiryayev (1977) Statistics of Random Processes 1—General Theory Springer-Verlag Berlin/Heidelberg/New York
R.S. Liptser A.N. Shiryayev (1978) Statistics of Random Processes 2—Applications Springer-Verlag Berlin/Heidelberg/New York
J.C. Maxwell (1871) Theory of Heat Longmans London
Mitter SK., Newton NJ. (2003). A Variational approach to non-linear estimation. SIAM J. Control Optim. 42:1813–1833
S.K. Mitter R.B. Vinter (1975) Filtering for linear stochastic hereditary differential systems, Lecture Notes in Economics and Mathematical Systems, 107 Control Theory, Numerical Methods and Computer Modelling Springer-Verlag Berlin/Heidelberg/New York 1–21
H. Nyquist (1928) ArticleTitleThermal agitation of electric charge in conductors Phys.Rev. 32 110–113
M.B. Propp (1985) The Thermodynamic Properties of Markov Processes, Ph.D.Thesis Massachusetts Institute of Technology Cambridge, MA, USA
D. Ruelle (1996) ArticleTitlePosititivity of entropy production in non-equilibrium statistical mechanics J.Stat.Phys. 85 1–23
E.Mayer-Wolf and M.Zakai, On a formula relating the Shannon information to the Fisher information for the filtering problem, in Filtering and Control of Random Processes, H. Korezlioglu, G. Mazziotto, and S. Szpirglas, eds., Lecture Notes in Control and Information Sciences 61 (Springer, 1984), pp.164–171
Y. Tomita S. Omatu T. Soeda (1976) ArticleTitleAn application of the information theory to filtering problem Inf.Sci. 11 13–27
JC. Willems (1972) ArticleTitleDissipative dynamical systems, part i: general theory Arch. Ratinal. Mech.Anal. 45 321–351
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Mitter, S.K., Newton, N.J. Information and Entropy Flow in the Kalman–Bucy Filter. J Stat Phys 118, 145–176 (2005). https://doi.org/10.1007/s10955-004-8781-9
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/s10955-004-8781-9