Skip to main content
SpringerLink
Log in

Ensemble Kalman filter implementations based on shrinkage covariance matrix estimation

  • Published:
Ocean Dynamics Aims and scope Submit manuscript

Abstract

This paper develops efficient ensemble Kalman filter (EnKF) implementations based on shrinkage covariance estimation. The forecast ensemble members at each step are used to estimate the background error covariance matrix via the Rao-Blackwell Ledoit and Wolf estimator, which has been specifically developed to approximate high-dimensional covariance matrices using a small number of samples. Two implementations are considered: in the EnKF full-space (EnKF-FS) approach, the assimilation process is performed in the model space, while the EnKF reduce-space (EnKF-RS) formulation performs the analysis in the subspace spanned by the ensemble members. In the context of EnKF-RS, additional samples are taken from the normal distribution described by the background ensemble mean and the estimated background covariance matrix, in order to increase the size of the ensemble and reduce the sampling error of the filter. This increase in the size of the ensemble is obtained without running the forward model. After the assimilation step, the additional samples are discarded and only the model-based ensemble members are propagated further. Methodologies to reduce the impact of spurious correlations and under-estimation of sample variances in the context of the EnKF-FS and EnKF-RS implementations are discussed. An adjoint-free four-dimensional extension of EnKF-RS is also discussed. Numerical experiments carried out with the Lorenz-96 model and a quasi-geostrophic model show that the use of shrinkage covariance matrix estimation can mitigate the impact of spurious correlations during the assimilation process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Anderson JL, Anderson SL (1999) A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon Weather Rev 127(12):2741–2758

    Article  Google Scholar 

  • Anderson JL (2012) Localization and sampling error correction in ensemble Kalman filter data assimilation. Mon Weather Rev 140(7):2359–2371

    Article  Google Scholar 

  • Benedetti A, Fisher M (2007) Background error statistics for aerosols. Q J R Meteorol Soc 133(623):391–405

    Article  Google Scholar 

  • Bickel DR, Padilla M (2014) A prior-free framework of coherent inference and its derivation of simple shrinkage estimators. J Stat Plan Infer 145:204–221

    Article  Google Scholar 

  • Buehner M (2005) Ensemble-derived stationary and flow-dependent background-error covariances: evaluation in a quasi-operational NWP setting. Q J R Meteorol Soc 131(607):1013–1043

    Article  Google Scholar 

  • Buehner M (2011) Evaluation of a spatial/spectral covariance localization approach for atmospheric data assimilation. Mon Weather Rev 140(2):617–636

    Article  Google Scholar 

  • Chatterjee A, Engelen RJ, Kawa SR, Sweeney C, Michalak AM (2013) Background error covariance estimation for atmospheric co2 data assimilation. J Geophys Res Atmos 118(17):10,140–10,154

    Article  Google Scholar 

  • Cheng H, Jardak M, Alexe M, Sandu A (2010) A hybrid approach to estimating error covariances in variational data assimilation. Tellus A 62(3):288–297

    Article  Google Scholar 

  • Couillet R, Matthew M (2014) Large dimensional analysis and optimization of robust shrinkage covariance matrix estimators. J Multivar Anal 131:99–120

    Article  Google Scholar 

  • Chen Y, Wiesel A, Eldar YC, Hero AO (2010) Shrinkage algorithms for MMSE covariance estimation. IEEE Trans Signal Process 58(10):5016–5029

    Article  Google Scholar 

  • Chen Y, Wiesel A, Hero AO (2011) Robust shrinkage estimation of high-dimensional covariance matrices. IEEE Trans Signal Process 59(9):4097–4107

    Article  Google Scholar 

  • Chen X, Wang ZJ, McKeown MJ (2012) Shrinkage-to-tapering estimation of large covariance matrices. IEEE Trans Signal Process 60(11):5640–5656

    Article  Google Scholar 

  • Cai TT, Zhang C-H, Zhou HH (2010) Optimal rates of convergence for covariance matrix estimation. Ann Stat 38(4):2118–2144

    Article  Google Scholar 

  • DeMiguel V, Martin-Utrera A, Nogales FJ. (2013) Size matters: optimal calibration of shrinkage estimators for portfolio selection. J Bank Financ 37(8):3018–3034

    Article  Google Scholar 

  • Evensen G (2003) The ensemble Kalman filter: theoretical formulation and practical implementation. Ocean Dyn 53(4): 343–367

    Article  Google Scholar 

  • Evensen G (2006) Data assimilation: the ensemble Kalman filter. Springer-Verlag New York, Inc., Secaucus

  • Elsheikh AH, Wheeler MF, Hoteit I (2013) An iterative stochastic ensemble method for parameter estimation of subsurface flow models. J Comput Phys 242:696–714

    Article  Google Scholar 

  • Farebrother RW (1978) A class of shrinkage estimators. J R Stat Soc Ser B Methodol 40(1):47–49

    Google Scholar 

  • Fisher TJ, Sun X (2011) Improved stein-type shrinkage estimators for the high-dimensional multivariate normal covariance matrix. Comput Stat Data Anal 55(5):1909–1918

    Article  Google Scholar 

  • Gillijns S, Mendoza OB, Chandrasekar J, De Moor BLR, Bernstein DS, Ridley A (2006) What is the ensemble Kalman filter and how well does it work?. In: American Control Conference, 2006 , p 6

  • Hoelzemann JJ, Elbern H, Ebel A (2001) PSAS and 4D-Var data assimilation for chemical state analysis by urban and rural observation sites. Phys Chem Earth B Hydrol Oceans Atmos 26(10):807–812

    Article  Google Scholar 

  • Hollingsworth A, Lonnberg P (1986) The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: the wind field. Tellus A 38A(2):111–136

    Article  Google Scholar 

  • Poterjoy J, Zhang F, Weng Y (2014) The effects of sampling errors on the EnKF assimilation of inner-core hurricane observations. Mon Weather Rev 142(4):1609–1630

    Article  Google Scholar 

  • Johnson CC, Jalali A, Ravikumar PD (2012) High-dimensional sparse inverse covariance estimation using greedy methods. In: Lawrence ND, Girolami MA (eds) Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics (AISTATS-12), vol 22, pp 574–582

  • Keppenne CL (2000) Data assimilation into a primitive-equation model with a parallel ensemble Kalman filter. Mon Weather Rev 128(6):1971–1981

    Article  Google Scholar 

  • Lermusiaux PFJ (2007) Adaptive modeling, adaptive data assimilation and adaptive sampling. Physica D: Nonlinear Phenomena 230:172–196

    Article  Google Scholar 

  • Lorenc AC (1986) Analysis methods for numerical weather prediction. Q J R Meteorol Soc 112(474):1177–1194

    Article  Google Scholar 

  • Lorenz EN (2005) Designing chaotic models. J Atmos Sci 62(5):1574–1587

    Article  Google Scholar 

  • Lermusiaux PFJ, Robinson AR (1999) Data assimilation via error subspace statistical estimation. part i: theory and schemes. Accessed: 08-29-2015

  • Ledoit O, Wolf M (2004) A well-conditioned estimator for large-dimensional covariance matrices. J Multivar Anal 88(2):365–411

    Article  Google Scholar 

  • Nino Ruiz ED, Sandu A, Anderson J (2014) An efficient implementation of the ensemble Kalman filter based on an iterative ShermanMorrison formula. Stat Comput:1–17

  • Ott E, Hunt BR, Szunyogh I, Zimin AV, Kostelich EJ, Corazza M, Kalnay E, Patil DJ, Yorke JA (2004) A local ensemble Kalman filter for atmospheric data assimilation. Tellus A 56(5):415–428

    Article  Google Scholar 

  • Park J (2014) Shrinkage estimator in normal mean vector estimation based on conditional maximum likelihood estimators. Stat Probab Lett 93:1–6

    Article  Google Scholar 

  • Ravikumar P, Wainwright MJ, Raskutti G, Yu B (2011) High-dimensional covariance estimation by minimizing L1-penalized log-determinant divergence. Electron J Stat 5:935–980

    Article  Google Scholar 

  • Sakov P, Bertino L (2011) Relation between two common localisation methods for the ENKF. Comput Geosci 15(2):225–237

    Article  Google Scholar 

  • Song H, Hoteit I, Cornuelle BD, Subramanian AC (2010) An adaptive approach to mitigate background covariance limitations in the ensemble Kalman filter. Mon Weather Rev 138(7):2825–2845

    Article  Google Scholar 

  • Sakov P, Oke P (2008) A deterministic formulation of the ensemble Kalman filter: an alternative to ensemble square root filters. Tellus A 60(2)

  • Whitaker JS, Hamill Thomas M (2002) Ensemble data assimilation without perturbed observations. Mon Weather Rev 16(3):1913–1924

    Article  Google Scholar 

  • Zupanski M (2009) Theoretical and practical issues of ensemble data assimilation in weather and climate. In: Park SK, Xu L (eds) Data assimilation for atmospheric, oceanic and hydrologic applications. Springer, Heidelberg, pp 67–84

Download references

Acknowledgments

This work was supported in part by awards NSF CCF–1218454, AFOSR FA9550–12–1–0293–DEF, and by the Computational Science Laboratory at Virginia Tech.

Author information

Authors and Affiliations

  1. Computational Science Laboratory, Department of Computer Science, Virginia Polytechnic Institute and State University, Blacksburg, VA, 24060, USA

    Elias D. Nino-Ruiz & Adrian Sandu

  2. Department of Computer Science, Universidad del Norte, Barranquilla, ATL, Colombia

    Elias D. Nino-Ruiz

Authors
  1. Elias D. Nino-Ruiz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adrian Sandu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elias D. Nino-Ruiz.

Additional information

Responsible Editor: Pierre Lermusiaux

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nino-Ruiz, E.D., Sandu, A. Ensemble Kalman filter implementations based on shrinkage covariance matrix estimation. Ocean Dynamics 65, 1423–1439 (2015). https://doi.org/10.1007/s10236-015-0888-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10236-015-0888-9

Keywords

Search

Navigation