Skip to main content
SpringerLink
Log in

A Maximum Entropy Method for Particle Filtering

  • Published:
Journal of Statistical Physics Aims and scope Submit manuscript

Abstract

Standard ensemble or particle filtering schemes do not properly represent states of low priori probability when the number of available samples is too small, as is often the case in practical applications. We introduce here a set of parametric resampling methods to solve this problem. Motivated by a general H-theorem for relative entropy, we construct parametric models for the filter distributions as maximum-entropy/minimum-information models consistent with moments of the particle ensemble. When the prior distributions are modeled as mixtures of Gaussians, our method naturally generalizes the ensemble Kalman filter to systems with highly non-Gaussian statistics. We apply the new particle filters presented here to two simple test cases: a one-dimensional diffusion process in a double-well potential and the three-dimensional chaotic dynamical system of Lorenz.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. R. V. Abramov and A. J. Majda, Quantifying uncertainty for non-gaussian ensembles in complex systems. Siam J. Set. Comp. 26:411–447 (2004).

    Article  MathSciNet  MATH  Google Scholar 

  2. D. L. Alspach and H. W. Sorenson, Nonlinear bayesian estimation using gaussian sum approximation. IEEE Trans. Auto. Cont. 17:439–448 (1972).

    Article  MATH  Google Scholar 

  3. J. L. Anderson, An ensemble adjustment filter for data assimilation. Mon. Wea. Rev. 129:2884–2903 (2001).

    Article  ADS  Google Scholar 

  4. J. L. Anderson and S. L. Anderson, A monte-carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Wea. Rev. 127:2741–2758 (1999).

    Article  ADS  Google Scholar 

  5. T. Bengtsson, C. Snyder, and D. Nychka, Toward a nonlinear ensemble filter for high-dimensional systems. J. Geophys. Res. 108:8775 (2003).

    Article  Google Scholar 

  6. A. F. Bennett, Inverse Methods in Physical Oceanography (Cambridge University Press, Cambridge, 1992).

    MATH  Google Scholar 

  7. G. Burgers, P. J. van Leeuwen, and G. Evensen, Analysis scheme in the ensemble kalman filter. Mon. Wea. Rev. 126:1719–1724 (1998).

    Article  ADS  Google Scholar 

  8. P. Cessi and W. R. Young, Multiple equilibria in two-dimensional thermo-haline circulation. J. Fluid Mech. 241:291–309 (1992).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  9. T. M. Cover and J. A. Thomas, Elements of Information Theory (John Wiley & Sons, 1991).

  10. D. Crisan and A. Doucet, A survey of convergence results on particle filtering for practitioners. IEEE Trans. Signal Process. 50:736–746 (2002).

    Article  MathSciNet  ADS  Google Scholar 

  11. L. Devroye, The double kernel method in density estimation. Ann. Inst. H. Poincarè 25:533–580 (1989).

    MathSciNet  MATH  Google Scholar 

  12. A. Doucet, N. de Freitas, and N. Gordon (eds.), Sequential Monte Carlo Pract (Springer-Verlag, 2001).

  13. A. W. F. Edwards, Likelihood (The Johns Hopkins University Press, Baltimore, MD, 1992).

    MATH  Google Scholar 

  14. M. Ehrendorfer, The liouville equation and its potential usefulness for the prediction of forecast skill, part i: Theory. Mon. Wea. Rev. 122:703–713 (1994).

    Article  ADS  Google Scholar 

  15. R. Ellis, Entropy, Large Deviations, and Statistical Mechanics (Springer-Verlag, New York, 1985).

    MATH  Google Scholar 

  16. G. Evensen, Inverse methods and data assimilation in nonlinear ocean modles. Physica D 77:108–129, (1994a).

    Article  ADS  Google Scholar 

  17. G. Evensen, Sequential data assimilation with a nonlinear quasigeostrophic model using monte carlo methods to forecast error statistics. J. Geophys. Res. 99 (C5):10,143–10,162, (1994b).

    Article  ADS  Google Scholar 

  18. G. Evensen, Sampling strategies and square-root analysis schemes for the enkf. Ocean Model. 54:539–560 (2004).

    ADS  Google Scholar 

  19. G. Eyink, Turbulence noise. J. Stat. Phys. 83:955–1019 (1996).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  20. G. L. Eyink, Statistical hydrodynamics of the thermohaline circulation in a two-dimensional model. Tellus A 57:100–115 (2005).

    Article  ADS  Google Scholar 

  21. G. L. Eyink, J. M. Restrepo, and F. J. Alexander, A mean field approximation in data assimilation for nonlinear dynamics. Physica D 95:347–368 (2004).

    Article  ADS  MathSciNet  Google Scholar 

  22. D. Fox, Adapting the sample size in particle filters through kid-sampling. Int. J. Robot. Res. 12:985–1003 (2003).

    Article  Google Scholar 

  23. M. Frontini and A. Tagliani, Maximum entropy in the finite stieltjes and hamburger moment problem. J. Math. Phys. 35:6748–6756 (1994).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  24. A. Gelb, Applied Optimal Estimation (The MIT Press, Cambridge, MA, 1974).

    Google Scholar 

  25. M. Ghil and K. Ide, Data assimilation in meteorology and oceanography: Theory and practice. J. Meteor. Soc. Japan 75:111–496 (1997).

    Google Scholar 

  26. A. Golan, G. Judge, and D. Miller, Maximum Entropy Econometrics: Robust Estimation with Limited Data (John Wiley & Sons, New York, 1996).

    MATH  Google Scholar 

  27. N. J. Gordon, D. J. Salmond, and A. F. M. Smith, Novel approach to nonlinear/non-gaussian bayesian state estimation. IEE Proc.-F 140:107–113 (1993).

    Google Scholar 

  28. A. Hannachi and A. O’Neill, Atmospheric multiple equilibria and non-gaussian behaviour in model simulations. Quart. J. R. Meteor. Soc. 127:939–958 (2001).

    Article  ADS  Google Scholar 

  29. K. Haven, A. J. Majda, and R. V. Abramov, Quantifying predictability through information theory: small sample estimation in a non-gaussian framework. J. Comp. Phys. 206:334–362 (2005).

    Article  ADS  MathSciNet  MATH  Google Scholar 

  30. P. Hanggi, P. Talkner, and M. Borkovec, Reaction-rate theory: fifty years after kramers. Rev. Mod. Phys. 62(2):251–341 (1990).

    Article  ADS  MathSciNet  Google Scholar 

  31. P. L. Houtekamer and H. L. Mitchell, A sequential ensemble kalman filter for atmospheric data assimilation. Mon. Wea. Rev. 129:123–137 (2001).

    Article  ADS  Google Scholar 

  32. L. F. James, C. E. Priebe, and D. J. Marchette, Consistent estimation of mixture complexity. Ann. Stat. 29(5):1281–1296 (2001).

    Article  MathSciNet  MATH  Google Scholar 

  33. A. H. Jazwinski, Stochastic Processes and Filtering Theory (Academic Press, NY, 1970).

    MATH  Google Scholar 

  34. C. L. Keppenne, Data assimilation into a primitive-equation model with a parallel ensemble kalman filter. Mon. Wea. Rev. 128:1971–1981 (2000).

    Article  ADS  Google Scholar 

  35. S. Kim, G. L. Eyink, J. M. Restrepo, F. J. Alexander, and G. Johnson, Ensemble filtering for nonlinear dynamics. Mon. Wea. Rev. 131:2586–2594 (2003).

    Article  ADS  Google Scholar 

  36. G. Kitagawa and W. Gersch, Smoothness Priors Analysis of Time Series, vol. 116 of Lecture Notes in Statistics (Springer-Verlag, 1996).

  37. R. Kleeman, Measuring dynamical prediction utility using relative entropy. J. Atmos. Sci. 59:2057–2072 (2002).

    Article  ADS  Google Scholar 

  38. P. E. Kloeden and E. Platen, Numerical Solution of Stochastic Differential Equations (Springer Verlag, 1997).

  39. S. Kullback and R. A. Leibler, On information and sufficiency. Ann. Math. Stat. 22:7986 (1951).

    Article  MathSciNet  Google Scholar 

  40. K. Lindenberg, B. J. West, and J. Kottalam, Fluctuations and dissipation in problems of geophysical fluid dynamics. In Irreversible Phenomena and Dynamical Systems Analysis in Geo-sciences, C. Nicolis and G. Nicolis (eds.) (NATO ASI, Series C. D. Reidel Publ. Co., 1987).

  41. M. Loève, Probability Theory, 3rd ed. (Van Nostrand, New York, 1963).

    MATH  Google Scholar 

  42. E. N. Lorenz, Deterministic nonperiodic flow. J. Atmos. Sci. 20:130–141 (1963).

    Article  ADS  Google Scholar 

  43. G. McLachlan and D. Peel, Finite Mixture Models (John Wiley & Sons, New York, 2000).

    MATH  Google Scholar 

  44. N. Metropolis and S. Ulam, The monte carlo method. J. Amer. Stat. Assoc. 44:335–341 (1949).

    Article  MathSciNet  MATH  Google Scholar 

  45. P. Del Moral, Nonlinear filtering: Interacting particle solution. Markov Proc. Rel. Fields. 2:555–579 (1996).

  46. P. Del Moral, Nonlinear filtering using random particles. Theor. Probab. Appl. 40:690–701 (1996).

    Article  MathSciNet  Google Scholar 

  47. P. Del Moral, J. Jacod, and Ph. Protter, The monte carlo method for filtering with discrete-time observations. Probab. Theory. Rel. 120:346–368 (2001).

    Article  MathSciNet  MATH  Google Scholar 

  48. J. Namias, The index cycle and its role in the general circulation. J. Meteorol. 7:130–139 (1950).

    Google Scholar 

  49. J. Nocedal and S. J. Wright, Numerical Optimization (Springer Series in Operations Research. Springer-Verlag, 1999).

  50. Y. Pawitan, In All Likelihood: Statistical Modelling and Inference Using Likelihood (Oxford University Press, Oxford, UK, 2001).

    MATH  Google Scholar 

  51. C. E. Priebe, Adaptive mixtures. J. Amer. Stat. Assoc. 89:796–806 (1994).

    Article  MathSciNet  MATH  Google Scholar 

  52. C. E. Priebe and D. J. Marchette, Alternating kernel and mixture density estimates. Comput. Stat. Data. An. 35:43–65 (2000).

    Article  MathSciNet  MATH  Google Scholar 

  53. D. Rex, Blocking action in the middle troposphere and its effect upon regional climate. Tellus 2:196–211 (1950).

    Google Scholar 

  54. H. Risken, The Fokker-Planck Equation (Springer-Verlag, New York, 1984).

    MATH  Google Scholar 

  55. F. Schoegl, Fluctuations in thermodynamic non-equilibrium states. Z. Physik. 244:199–205 (1971).

    Article  ADS  Google Scholar 

  56. F. Schoegl, On stability of steady states. Z. Physik 243:303–310 (1971).

    Article  ADS  Google Scholar 

  57. D. W. Scott, Multivariate Density Estimation (John Wiley, New York, 1992).

    MATH  Google Scholar 

  58. B. W. Silverman, Density Estimation for Statistics and Data Analysis (Chapman and Hall, New York, 1986).

    MATH  Google Scholar 

  59. P. Smyth, Model selection for probabilistic clustering using cross-validated likelihood. Stat. Comput. 10:63–72 (2000).

    Article  Google Scholar 

  60. P. Smyth, K. Ide, and M. Ghil, Multiple regimes in northern hemisphere height fields via mixture model clustering. J. Atmos. Sci. 56:3704–3723 (1999).

    Article  ADS  Google Scholar 

  61. D. B. Stephenson, A. Hannachi, and A. O Neill, On the existence of multiple climate regimes. Q. J. R. Meteor. Soc. 130:583–605 (2004).

    Article  ADS  Google Scholar 

  62. H. Stommel, Thermohaline convection with two stable regimes of flow. Tellus 13:224–230 (1961).

    Article  Google Scholar 

  63. A. Tarantola, Inverse Problem Theory (Elsevier Science, 1987).

  64. H. Theil and D. G. Fiebig, Exploiting Continuity: Maximum Entropy Estimation of Continuous Distributions (Ballinger Publishing Co., Cambridge, MA, 1984).

    Google Scholar 

  65. W. Tucker, A rigorous ode solver and smale’s 14th problem. Found. Comput. Math. 2:53–117 (2002).

    MathSciNet  MATH  Google Scholar 

  66. P. J. van Leeuwen and G. Evensen, Data assimilation and inverse methods in terms of a probabilistic formulation. Mon. Wea. Rev. 124:2898–2913 (1996).

    Article  ADS  Google Scholar 

  67. M. P. Wand and M. C. Jones, Kernel Smoothing (Chapman and Hall, London, 1995).

    MATH  Google Scholar 

  68. L.-S. Young, What are srb measures and which dynamical systems have them? J. Stat. Phys. 108:733–754 (2002).

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Applied Mathematics and Statistics, The Johns Hopkins University, Baltimore, MD, 21218, U.S.A.

    Gregory L. Eyink

  2. Department of Mathematics, University of Arizona, Tucson, AZ, 85721, U.S.A.

    Sangil Kim

Authors
  1. Gregory L. Eyink
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sangil Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gregory L. Eyink.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Eyink, G.L., Kim, S. A Maximum Entropy Method for Particle Filtering. J Stat Phys 123, 1071–1128 (2006). https://doi.org/10.1007/s10955-006-9124-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10955-006-9124-9

Keywords

Search

Navigation