Skip to main content
SpringerLink
Log in

An efficient implementation of the ensemble Kalman filter based on an iterative Sherman–Morrison formula

  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

We present a practical implementation of the ensemble Kalman filter (EnKF) based on an iterative Sherman–Morrison formula. The new direct method exploits the special structure of the ensemble-estimated error covariance matrices in order to efficiently solve the linear systems involved in the analysis step of the EnKF. The computational complexity of the proposed implementation is equivalent to that of the best EnKF implementations available in the literature when the number of observations is much larger than the number of ensemble members, as typically is case in practice. Moreover, the proposed method provides the best theoretical complexity when it is compared to generic formulations of matrix inversion based on the Sherman–Morrison formula. The stability analysis of the proposed method is carried out and a pivoting strategy is discussed in order to reduce the accumulation of round-off errors without increasing the computational effort. A parallel implementation is discussed as well. Computational experiments carried out using an oceanic quasi-geostrophic model reveal that the proposed algorithm yields the same accuracy as other EnKF implementations, but scales better with regard to the number of observations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Alves, O., Robert, C.: Tropical Pacific Ocean model error covariances from Monte Carlo simulations. Q. J. R. Meteorol. Soc. 131, 3643–3658 (2005)

    Article  Google Scholar 

  • Anderson, J.L.: An ensemble adjustment Kalman filter for data assimilation. Mon. Weather Rev. 129(12), 2884–2903 (2001)

    Article  Google Scholar 

  • Anderson, J.L.: Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter. Physica D Nonlinear Phenomena 230(1–2), 99–111 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  • Anderson, J.L., Collins, N.: Scalable implementations of ensemble filter algorithms for data assimilation. J. Atmos. Ocean. Technol. 24(8), 1452–1463 (2007)

    Article  Google Scholar 

  • Angerson, E., Bai, Z., Dongarra, J., Greenbaum, A., McKenney, A., Du Croz, J., Hammarling, S., Demmel, J., Bischof, C., Sorensen, D.: Lapack: A portable linear algebra library for high-performance computers. In: Proceedings of Supercomputing 1990. pp. 2–11 (1990). doi:10.1109/SUPERC.1990.129995

  • Carton, X., Baraille, R.: Data assimilation in quasi-geostrophic ocean models. In: Proceedings of OCEANS 94. Oceans Engineering for Today’s Technology and Tomorrow’s Preservation. vol. 3, pp. III/337–III/346 (1994). doi:10.1109/OCEANS.1994.364221

  • Cohen, A.: Rate of convergence of several conjugate gradient algorithms. SIAM J. Numer. Anal. 9(2), 248–259 (1972)

    Article  MATH  MathSciNet  Google Scholar 

  • Eisenstat, S.: Efficient implementation of a class of preconditioned conjugate gradient methods. SIAM J. Sci. Stat. Comput. 2(1), 1–4 (1981)

    Article  MATH  MathSciNet  Google Scholar 

  • Evensen, G.: Chap. 14. Data Assimilation: The Ensemble Kalman Filter, 2nd edn. Springer, New York (2009)

    Chapter  Google Scholar 

  • Evensen, G.: Estimation in an oil reservoir simulator. Data Assimilation, pp. 263–272. Springer, Berlin (2009)

    Chapter  Google Scholar 

  • Evensen, G.: The ensemble Kalman filter for combined state and parameter estimation. Control Syst. IEEE 29(3), 83–104 (2009)

    Article  MathSciNet  Google Scholar 

  • Fraser, A.M.: Hidden Markov models and dynamical systems. SIAM (2008). doi:10.1137/1.9780898717747

  • Gill, P., Saunders, M., Shinnerl, J.: On the stability of Cholesky factorization for symmetric quasidefinite systems. SIAM J. Matrix Anal. Appl. 17(1), 35–46 (1996)

    Article  MATH  MathSciNet  Google Scholar 

  • Godinez, H., Moulton, J.: An efficient matrix-free algorithm for the ensemble Kalman filter. Comput. Geosci. 16(3), 565–575 (2012). doi:10.1007/s10596-011-9268-9

    Article  Google Scholar 

  • Golub, G., OLeary, D.: Some history of the conjugate gradient and Lanczos algorithms: 1948–1976. SIAM Rev. 31(1), 50–102 (1989)

    Article  MATH  MathSciNet  Google Scholar 

  • Haugen, V., Naevdal, G., Natvik, L.J., Evensen, G., Berg, A., Flornes, K.: History matching using the ensemble Kalman filter on a north sea field case. SPE J. 13, 382–391 (2008)

    Article  Google Scholar 

  • Kovalenko, A., Mannseth, T., Nævdal, G.: Error estimate for the ensemble Kalman filter analysis step. SIAM J. Matrix Anal. Appl. 32(4), 1275–1287 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  • Li, H., Kalnay, E., Miyoshi, T.: Simultaneous estimation of covariance inflation and observation errors within an ensemble Kalman filter. Q. J. R. Meteorol. Soc. 135(639), 523–533 (2009). doi:10.1002/qj.371

    Article  Google Scholar 

  • Lin, C., More, J.: Incomplete Cholesky factorizations with limited memory. SIAM J. Sci. Comput. 21(1), 24–45 (1999). doi:10.1137/S1064827597327334

    Article  MATH  MathSciNet  Google Scholar 

  • Mandel, J.: Efficient implementation of the ensemble Kalman filter. University of Colorado at Denver and Health Sciences Center, Technical Report (2006)

  • Maponi, P.: The solution of linear systems by using the Sherman–Morrison formula. Linear Algebra Appl. 420(2–3), 276–294 (2007). doi:10.1016/j.laa.2006.07.007

    Article  MATH  MathSciNet  Google Scholar 

  • Meinguet, J.: Refined error analyses of Cholesky factorization. SIAM J. Numer. Anal. 20(6), 1243–1250 (1983)

    Google Scholar 

  • Ott, E., Hunt, B.R., Szunyogh, I., Zimin, A.V., Kostelich, E.J., Corazza, M., Kalnay, E., Patil, D.J., Yorke, J.A.: A local ensemble Kalman filter for atmospheric data assimilation. Tellus A 56(5), 415–428 (2004)

    Google Scholar 

  • Otto, K., Larsson, E.: Iterative solution of the Helmholtz equation by a second-order method. SIAM J. Matrix Anal. Appl. 21(1), 209–229 (1999). doi:10.1137/S0895479897316588

    Article  MATH  MathSciNet  Google Scholar 

  • Reid, J.: The use of conjugate gradients for systems of linear equations possessing property A. SIAM J. Numer. Anal. 9(2), 325–332 (1972)

    Article  MATH  MathSciNet  Google Scholar 

  • Schnabel, R., Eskow, E.: A new modified Cholesky factorization. SIAM J. Sci. Stat. Comput. 11(6), 1136–1158 (1990). doi:10.1137/0911064

    Article  MATH  MathSciNet  Google Scholar 

  • Stewart, M., Van Dooren, P.: Stability issues in the factorization of structured matrices. SIAM J. Matrix Anal. Appl. 18(1), 104–118 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  • Suarez, A., Heather Dawn, R., Dustan, W., Coniglio, M.: Comparison of ensemble Kalman filter-based forecasts to traditional ensemble and deterministic forecasts for a case study of banded snow. Weather Forecast. 27, 85–105 (2012)

    Article  Google Scholar 

  • Tippett, M.K., Anderson, J.L., Bishop, C.H., Hamill, T.M., Whitaker, J.S.: Ensemble square root filters. Mon. Weather Rev. 131(7), 1485–1490 (2003)

    Article  Google Scholar 

  • Wu, G., Zheng, X., Li, Y.: Inflation adjustment on error covariance matrix of ensemble Kalman filter. In: International Conference on Multimedia Technology (ICMT), 2011. pp. 2160–2163 (2011)

  • Xia, J., Gu, M.: Robust approximate Cholesky factorization of rank-structured symmetric positive definite matrices. SIAM J. Matrix Anal. Appl. 31(5), 2899–2920 (2010). doi:10.1137/090750500

    Article  MATH  MathSciNet  Google Scholar 

Download references

Acknowledgments

This work has been supported in part by NSF through awards NSF OCI-8670904397, NSF CCF-0916493, NSF DMS-0915047, NSF CMMI-1130667, NSF CCF-1218454, AFOSR FA9550-12-1-0293-DEF, AFOSR 12-2640-06, and by the Computational Science Laboratory at Virginia Tech.

Author information

Authors and Affiliations

  1. Computational Science Laboratory, Department of Computer Science, Virginia Polytechnic Institute and State University, Blacksburg, VA , 24060, USA

    Elias D. Nino Ruiz & Adrian Sandu

  2. Department of Computer Science, Universidad del Norte, Barranquilla, AT, Colombia

    Elias D. Nino Ruiz

  3. Data Assimilation Research Section, Institute for Mathematics Applied to Geosciences, National Center for Atmospheric Research, Boulder, CO , 80307-3000, USA

    Jeffrey Anderson

Authors
  1. Elias D. Nino Ruiz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adrian Sandu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeffrey Anderson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elias D. Nino Ruiz.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nino Ruiz, E.D., Sandu, A. & Anderson, J. An efficient implementation of the ensemble Kalman filter based on an iterative Sherman–Morrison formula. Stat Comput 25, 561–577 (2015). https://doi.org/10.1007/s11222-014-9454-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-014-9454-4

Keywords

Search

Navigation