Skip to main content
SpringerLink
Log in

An adjoint-free four-dimensional variational data assimilation method via a modified Cholesky decomposition and an iterative Woodbury matrix formula

  • Original paper
  • Published:
Nonlinear Dynamics Aims and scope Submit manuscript

Abstract

In this paper, we propose an efficient and practical implementation of a four-dimensional variational ensemble Kalman filter (4D-EnKF) via a modified Cholesky decomposition. The main scope of our method is to avoid the intrinsic needed of adjoint models in the four-dimensional context. As it is well known, in practice, adjoint models can be labor-intensive to develop and computationally expensive to run. We avoid the use of adjoint models by taking snapshots of an ensemble of model realizations at observation times. Then, we employ a modified Cholesky decomposition on those ensembles to build control spaces, which in turn are employed to estimate analysis increments and to mitigate the impact of sampling noise. We discuss a matrix-free implementation of our 4D-EnKF formulation via the Woodbury matrix identity. Experimental tests are performed by using the Lorenz 96 model and an atmospheric general circulation model. The numerical results reveal that the accuracy of our proposed filter implementation outperforms those of traditional 4D-EnKF formulations in terms of L-2 error norms and root-mean-square error values.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Anderson, J.L.: An ensemble adjustment Kalman filter for data assimilation. Mon. Weather Rev. 129(12), 2884–2903 (2001)

    Google Scholar 

  2. Anderson, J.L.: A nonlinear rank regression method for ensemble Kalman filter data assimilation. Mon. Weather Rev. 147(8), 2847–2860 (2019)

    Google Scholar 

  3. Bannister, R.: A review of operational methods of variational and ensemble-variational data assimilation. Q. J. R. Meteorol. Soc. 143(703), 607–633 (2017)

    Google Scholar 

  4. Benedetti, A., Stephens, G.L., Vukićević, T.: Variational assimilation of radar reflectivities in a cirrus model. I: model description and adjoint sensitivity studies. Q. J. R. Meteorol. Soc. J. Atmos. Sci. Appl. Meteorol. Phys. Oceanogr. 129(587), 277–300 (2003)

    Google Scholar 

  5. Bickel, P.J., Levina, E., et al.: Covariance regularization by thresholding. Ann. Stat. 36(6), 2577–2604 (2008)

    Google Scholar 

  6. Bickel, P.J., Levina, E., et al.: Regularized estimation of large covariance matrices. Ann. Stat. 36(1), 199–227 (2008)

    Google Scholar 

  7. Bracco, A., Kucharski, F., Kallummal, R., Molteni, F.: Internal variability, external forcing and climate trends in multi-decadal AGCM ensembles. Clim. Dyn. 23(6), 659–678 (2004)

    Google Scholar 

  8. Burgers, G., Jan van Leeuwen, P., Evensen, G.: Analysis scheme in the ensemble Kalman filter. Mon. Weather Rev. 126(6), 1719–1724 (1998)

    Google Scholar 

  9. Chang, F.J., Chiang, Y.M., Tsai, M.J., Shieh, M.C., Hsu, K.L., Sorooshian, S.: Watershed rainfall forecasting using neuro-fuzzy networks with the assimilation of multi-sensor information. J. Hydrol. 508, 374–384 (2014)

    Google Scholar 

  10. Chen, Y., Oliver, D.S.: Cross-covariances and localization for enkf in multiphase flow data assimilation. Comput. Geosci. 14(4), 579–601 (2010)

    Google Scholar 

  11. Dormand, J.R., Prince, P.J.: A family of embedded Runge–Kutta formulae. J. Comput. Appl. Math. 6(1), 19–26 (1980)

    Google Scholar 

  12. Evensen, G.: The ensemble Kalman filter: theoretical formulation and practical implementation. Ocean Dyn. 53(4), 343–367 (2003)

    Google Scholar 

  13. Fertig, E.J., Harlim, J., Hunt, B.R.: A comparative study of 4D-Var and a 4D ensemble Kalman filter: perfect model simulations with Lorenz-96. Tellus A 59(1), 96–100 (2007)

    Google Scholar 

  14. Godinez, H.C., Moulton, J.D.: An efficient matrix-free algorithm for the ensemble Kalman filter. Comput. Geosci. 16(3), 565–575 (2012)

    Google Scholar 

  15. Gottwald, G.A., Melbourne, I.: Testing for chaos in deterministic systems with noise. Phys. D Nonlinear Phenom. 212(1), 100–110 (2005)

    Google Scholar 

  16. Greybush, S.J., Kalnay, E., Miyoshi, T., Ide, K., Hunt, B.R.: Balance and ensemble Kalman filter localization techniques. Mon. Weather Rev. 139(2), 511–522 (2011)

    Google Scholar 

  17. Gustafsson, N.: Discussion on 4D-Var or ENKF? Tellus A Dyn. Meteorol. Oceanogr. 59(5), 774–777 (2007)

    Google Scholar 

  18. Gustafsson, N., Bojarova, J.: Four-dimensional ensemble variational (4d-En-Var) data assimilation for the high resolution limited area model (HIRLAM). Nonlinear Process. Geophys. 21(4), 745–762 (2014)

    Google Scholar 

  19. Han, Y., Zhang, J., Sun, D.: Error control and adjustment method for underwater wireless sensor network localization. Appl. Acoust. 130, 293–299 (2018)

    Google Scholar 

  20. Harlim, J., Hunt, B.R.: Four-dimensional local ensemble transform Kalman filter: numerical experiments with a global circulation model. Tellus A Dyn. Meteorol. Oceanogr. 59(5), 731–748 (2007)

    Google Scholar 

  21. Houtekamer, P.L., Mitchell, H.L.: Data assimilation using an ensemble Kalman filter technique. Mon. Weather Rev. 126(3), 796–811 (1998)

    Google Scholar 

  22. Huang, X.Y., Xiao, Q., Barker, D.M., Zhang, X., Michalakes, J., Huang, W., Henderson, T., Bray, J., Chen, Y., Ma, Z., et al.: Four-dimensional variational data assimilation for WRF: formulation and preliminary results. Mon. Weather Rev. 137(1), 299–314 (2009)

    Google Scholar 

  23. Ito, Si, Nagao, H., Yamanaka, A., Tsukada, Y., Koyama, T., Kano, M., Inoue, J.: Data assimilation for massive autonomous systems based on a second-order adjoint method. Phys. Rev. E 94(4), 043307 (2016)

    Google Scholar 

  24. Kalnay, E.: Atmospheric Modeling, Data Assimilation and Predictability. Cambridge University Press, Cambridge (2003)

    Google Scholar 

  25. Karimi, A., Paul, M.R.: Extensive Chaos in the Lorenz-96 model. Chaos Interdiscip. J. Nonlinear Sci. 20(4), 043105 (2010)

    Google Scholar 

  26. Kucharski, F., Molteni, F., Bracco, A.: Decadal interactions between the western tropical Pacific and the North Atlantic oscillation. Clim. Dyn. 26(1), 79–91 (2006)

    Google Scholar 

  27. Lahoz, B.K.W., Menard, R.: Data Assimilation. Springer, Berlin (2010)

    Google Scholar 

  28. Lei, L., Whitaker, J.S., Bishop, C.: Improving assimilation of radiance observations by implementing model space localization in an ensemble Kalman filter. J. Adv. Model. Earth Syst. 10(12), 3221–3232 (2018)

    Google Scholar 

  29. Levina, E., Rothman, A., Zhu, J., et al.: Sparse estimation of large covariance matrices via a nested Lasso penalty. Ann. Appl. Stat. 2(1), 245–263 (2008)

    Google Scholar 

  30. Liu, C., Xiao, Q., Wang, B.: An ensemble-based four-dimensional variational data assimilation scheme. Part I: technical formulation and preliminary test. Mon. Weather Rev. 136(9), 3363–3373 (2008)

    Google Scholar 

  31. Lorenc, A.C.: The potential of the ensemble Kalman filter for NWPA comparison with 4D-Var. Q. J. R. Meteorol. Soc. J. Atmos. Sci. Appl. Meteorol. Phys. Oceanogr. 129(595), 3183–3203 (2003)

    Google Scholar 

  32. Lorenc, A.C., Bowler, N.E., Clayton, A.M., Pring, S.R., Fairbairn, D.: Comparison of hybrid-4DEnVar and hybrid-4DVar data assimilation methods for global NWP. Mon. Weather Rev. 143(1), 212–229 (2015)

    Google Scholar 

  33. Lorenz, E.N.: Designing chaotic models. J. Atmos. Sci. 62(5), 1574–1587 (2005). https://doi.org/10.1175/JAS3430.1

    Google Scholar 

  34. Mandel, J., Bennethum, L.S., Beezley, J.D., Coen, J.L., Douglas, C.C., Kim, M., Vodacek, A.: A wildland fire model with data assimilation. Math. Comput. Simul. 79(3), 584–606 (2008)

    Google Scholar 

  35. Margvelashvili, N., Campbell, E.: Sequential data assimilation in fine-resolution models using error-subspace emulators: theory and preliminary evaluation. J. Mar. Syst. 90(1), 13–22 (2012)

    Google Scholar 

  36. Miyoshi, T.: The Gaussian approach to adaptive covariance inflation and its implementation with the local ensemble transform Kalman filter. Mon. Weather Rev. 139(5), 1519–1535 (2011)

    Google Scholar 

  37. Miyoshi, T., Kunii, M.: The local ensemble transform Kalman filter with the weather research and forecasting model: experiments with real observations. Pure Appl. Geophys. 169(3), 321–333 (2012)

    Google Scholar 

  38. Molteni, F.: Atmospheric simulations using a GCM with simplified physical parametrizations. I: model climatology and variability in multi-decadal experiments. Clim. Dyn. 20(2–3), 175–191 (2003)

    Google Scholar 

  39. Nerger, L., Schulte, S., Bunse-Gerstner, A.: On the influence of model nonlinearity and localization on ensemble Kalman smoothing. Q. J. R. Meteorol. Soc. 140(684), 2249–2259 (2014)

    Google Scholar 

  40. Nino-Ruiz, E.: A matrix-free posterior ensemble Kalman filter implementation based on a modified Cholesky decomposition. Atmosphere 8(7), 125 (2017)

    Google Scholar 

  41. Nino-Ruiz, E.D., Sandu, A., Anderson, J.: An efficient implementation of the ensemble Kalman filter based on an iterative Sherman–Morrison formula. Stat. Comput. 25(3), 561–577 (2015)

    Google Scholar 

  42. Nino-Ruiz, E.D., Sandu, A., Deng, X.: A parallel implementation of the ensemble Kalman filter based on modified Cholesky decomposition. J. Comput. Sci. 36, 100654 (2019). https://doi.org/10.1016/j.jocs.2017.04.005

    Google Scholar 

  43. Nino-Ruiz, E.D., Sandu, A., Deng, X.: An ensemble Kalman filter implementation based on modified Cholesky decomposition for inverse covariance matrix estimation. SIAM J. Sci. Comput. 40(2), A867–A886 (2018)

    Google Scholar 

  44. Reichle, R.H.: Data assimilation methods in the earth sciences. Adv. Water Resour. 31(11), 1411–1418 (2008)

    Google Scholar 

  45. Rothman, A.J., Levina, E., Zhu, J.: Generalized thresholding of large covariance matrices. J. Am. Stat. Assoc. 104(485), 177–186 (2009)

    Google Scholar 

  46. Ruiz, E.D.N., Sandu, A.: A derivative-free trust region framework for variational data assimilation. J. Comput. Appl. Math. 293, 164–179 (2016)

    Google Scholar 

  47. Shi, K., Wang, J., Tang, Y., Zhong, S.: Reliable asynchronous sampled-data filtering of T–S fuzzy uncertain delayed neural networks with stochastic switched topologies. Fuzzy Sets Syst. (2018). https://doi.org/10.1016/j.fss.2018.11.017

  48. Stengel, M., Undén, P., Lindskog, M., Dahlgren, P., Gustafsson, N., Bennartz, R.: Assimilation of SEVIRI infrared radiances with HIRLAM 4D-Var. Q. J. R. Meteorol. Soc. J. Atmos. Sci. Appl. Meteorol. Phys. Oceanogr. 135(645), 2100–2109 (2009)

    Google Scholar 

  49. Stroud, J.R., Katzfuss, M., Wikle, C.K.: A Bayesian adaptive ensemble Kalman filter for sequential state and parameter estimation. Mon. Weather Rev. 146(1), 373–386 (2018)

    Google Scholar 

  50. Tr’emolet, Y.: Accounting for an imperfect model in 4D-Var. Q. J. R. Meteorol. Soc. J. Atmos. Sci. Appl. Meteorol. Phys. Oceanogr. 132(621), 2483–2504 (2006)

    Google Scholar 

  51. Trémolet, Y.: Incremental 4D-Var convergence study. Tellus A Dyn. Meteorol. Oceanogr. 59(5), 706–718 (2007)

    Google Scholar 

  52. Wang, X., Hamill, T.M., Whitaker, J.S., Bishop, C.H.: A comparison of hybrid ensemble transform Kalman filter-optimum interpolation and ensemble square root filter analysis schemes. Mon. Weather Rev. 135(3), 1055–1076 (2007)

    Google Scholar 

  53. Yin, J., Zhan, X., Zheng, Y., Hain, C.R., Liu, J., Fang, L.: Optimal ensemble size of ensemble Kalman filter in sequential soil moisture data assimilation. Geophys. Res. Lett. 42(16), 6710–6715 (2015)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by award UN 2018-38 and by the Applied Math and Computer Science Lab at Universidad del Norte, Colombia.

Author information

Authors and Affiliations

  1. Applied Math and Computer Science Laboratory, Department of Computer Science, Universidad del Norte, Barranquilla, 080001, Colombia

    Elias D. Nino-Ruiz, Luis G. Guzman-Reyes & Rolando Beltran-Arrieta

Authors
  1. Elias D. Nino-Ruiz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luis G. Guzman-Reyes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rolando Beltran-Arrieta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elias D. Nino-Ruiz.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nino-Ruiz, E.D., Guzman-Reyes, L.G. & Beltran-Arrieta, R. An adjoint-free four-dimensional variational data assimilation method via a modified Cholesky decomposition and an iterative Woodbury matrix formula. Nonlinear Dyn 99, 2441–2457 (2020). https://doi.org/10.1007/s11071-019-05411-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11071-019-05411-w

Keywords

Mathematics Subject Classification

Search

Navigation