Ocean Dynamics

, Volume 65, Issue 11, pp 1423–1439 | Cite as

Ensemble Kalman filter implementations based on shrinkage covariance matrix estimation

  • Elias D. Nino-RuizEmail author
  • Adrian Sandu


This paper develops efficient ensemble Kalman filter (EnKF) implementations based on shrinkage covariance estimation. The forecast ensemble members at each step are used to estimate the background error covariance matrix via the Rao-Blackwell Ledoit and Wolf estimator, which has been specifically developed to approximate high-dimensional covariance matrices using a small number of samples. Two implementations are considered: in the EnKF full-space (EnKF-FS) approach, the assimilation process is performed in the model space, while the EnKF reduce-space (EnKF-RS) formulation performs the analysis in the subspace spanned by the ensemble members. In the context of EnKF-RS, additional samples are taken from the normal distribution described by the background ensemble mean and the estimated background covariance matrix, in order to increase the size of the ensemble and reduce the sampling error of the filter. This increase in the size of the ensemble is obtained without running the forward model. After the assimilation step, the additional samples are discarded and only the model-based ensemble members are propagated further. Methodologies to reduce the impact of spurious correlations and under-estimation of sample variances in the context of the EnKF-FS and EnKF-RS implementations are discussed. An adjoint-free four-dimensional extension of EnKF-RS is also discussed. Numerical experiments carried out with the Lorenz-96 model and a quasi-geostrophic model show that the use of shrinkage covariance matrix estimation can mitigate the impact of spurious correlations during the assimilation process.


EnKF Shrinkage covariance estimation Background errors Square root filter 



This work was supported in part by awards NSF CCF–1218454, AFOSR FA9550–12–1–0293–DEF, and by the Computational Science Laboratory at Virginia Tech.


  1. Anderson JL, Anderson SL (1999) A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon Weather Rev 127(12):2741–2758CrossRefGoogle Scholar
  2. Anderson JL (2012) Localization and sampling error correction in ensemble Kalman filter data assimilation. Mon Weather Rev 140(7):2359–2371CrossRefGoogle Scholar
  3. Benedetti A, Fisher M (2007) Background error statistics for aerosols. Q J R Meteorol Soc 133(623):391–405CrossRefGoogle Scholar
  4. Bickel DR, Padilla M (2014) A prior-free framework of coherent inference and its derivation of simple shrinkage estimators. J Stat Plan Infer 145:204–221CrossRefGoogle Scholar
  5. Buehner M (2005) Ensemble-derived stationary and flow-dependent background-error covariances: evaluation in a quasi-operational NWP setting. Q J R Meteorol Soc 131(607):1013–1043CrossRefGoogle Scholar
  6. Buehner M (2011) Evaluation of a spatial/spectral covariance localization approach for atmospheric data assimilation. Mon Weather Rev 140(2):617–636CrossRefGoogle Scholar
  7. Chatterjee A, Engelen RJ, Kawa SR, Sweeney C, Michalak AM (2013) Background error covariance estimation for atmospheric co2 data assimilation. J Geophys Res Atmos 118(17):10,140–10,154CrossRefGoogle Scholar
  8. Cheng H, Jardak M, Alexe M, Sandu A (2010) A hybrid approach to estimating error covariances in variational data assimilation. Tellus A 62(3):288–297CrossRefGoogle Scholar
  9. Couillet R, Matthew M (2014) Large dimensional analysis and optimization of robust shrinkage covariance matrix estimators. J Multivar Anal 131:99–120CrossRefGoogle Scholar
  10. Chen Y, Wiesel A, Eldar YC, Hero AO (2010) Shrinkage algorithms for MMSE covariance estimation. IEEE Trans Signal Process 58(10):5016–5029CrossRefGoogle Scholar
  11. Chen Y, Wiesel A, Hero AO (2011) Robust shrinkage estimation of high-dimensional covariance matrices. IEEE Trans Signal Process 59(9):4097–4107CrossRefGoogle Scholar
  12. Chen X, Wang ZJ, McKeown MJ (2012) Shrinkage-to-tapering estimation of large covariance matrices. IEEE Trans Signal Process 60(11):5640–5656CrossRefGoogle Scholar
  13. Cai TT, Zhang C-H, Zhou HH (2010) Optimal rates of convergence for covariance matrix estimation. Ann Stat 38(4):2118–2144CrossRefGoogle Scholar
  14. DeMiguel V, Martin-Utrera A, Nogales FJ. (2013) Size matters: optimal calibration of shrinkage estimators for portfolio selection. J Bank Financ 37(8):3018–3034CrossRefGoogle Scholar
  15. Evensen G (2003) The ensemble Kalman filter: theoretical formulation and practical implementation. Ocean Dyn 53(4): 343–367CrossRefGoogle Scholar
  16. Evensen G (2006) Data assimilation: the ensemble Kalman filter. Springer-Verlag New York, Inc., SecaucusGoogle Scholar
  17. Elsheikh AH, Wheeler MF, Hoteit I (2013) An iterative stochastic ensemble method for parameter estimation of subsurface flow models. J Comput Phys 242:696–714CrossRefGoogle Scholar
  18. Farebrother RW (1978) A class of shrinkage estimators. J R Stat Soc Ser B Methodol 40(1):47–49Google Scholar
  19. Fisher TJ, Sun X (2011) Improved stein-type shrinkage estimators for the high-dimensional multivariate normal covariance matrix. Comput Stat Data Anal 55(5):1909–1918CrossRefGoogle Scholar
  20. Gillijns S, Mendoza OB, Chandrasekar J, De Moor BLR, Bernstein DS, Ridley A (2006) What is the ensemble Kalman filter and how well does it work?. In: American Control Conference, 2006 , p 6Google Scholar
  21. Hoelzemann JJ, Elbern H, Ebel A (2001) PSAS and 4D-Var data assimilation for chemical state analysis by urban and rural observation sites. Phys Chem Earth B Hydrol Oceans Atmos 26(10):807–812CrossRefGoogle Scholar
  22. Hollingsworth A, Lonnberg P (1986) The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: the wind field. Tellus A 38A(2):111–136CrossRefGoogle Scholar
  23. Poterjoy J, Zhang F, Weng Y (2014) The effects of sampling errors on the EnKF assimilation of inner-core hurricane observations. Mon Weather Rev 142(4):1609–1630CrossRefGoogle Scholar
  24. Johnson CC, Jalali A, Ravikumar PD (2012) High-dimensional sparse inverse covariance estimation using greedy methods. In: Lawrence ND, Girolami MA (eds) Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics (AISTATS-12), vol 22, pp 574–582Google Scholar
  25. Keppenne CL (2000) Data assimilation into a primitive-equation model with a parallel ensemble Kalman filter. Mon Weather Rev 128(6):1971–1981CrossRefGoogle Scholar
  26. Lermusiaux PFJ (2007) Adaptive modeling, adaptive data assimilation and adaptive sampling. Physica D: Nonlinear Phenomena 230:172–196CrossRefGoogle Scholar
  27. Lorenc AC (1986) Analysis methods for numerical weather prediction. Q J R Meteorol Soc 112(474):1177–1194CrossRefGoogle Scholar
  28. Lorenz EN (2005) Designing chaotic models. J Atmos Sci 62(5):1574–1587CrossRefGoogle Scholar
  29. Lermusiaux PFJ, Robinson AR (1999) Data assimilation via error subspace statistical estimation. part i: theory and schemes. Accessed: 08-29-2015Google Scholar
  30. Ledoit O, Wolf M (2004) A well-conditioned estimator for large-dimensional covariance matrices. J Multivar Anal 88(2):365–411CrossRefGoogle Scholar
  31. Nino Ruiz ED, Sandu A, Anderson J (2014) An efficient implementation of the ensemble Kalman filter based on an iterative ShermanMorrison formula. Stat Comput:1–17Google Scholar
  32. Ott E, Hunt BR, Szunyogh I, Zimin AV, Kostelich EJ, Corazza M, Kalnay E, Patil DJ, Yorke JA (2004) A local ensemble Kalman filter for atmospheric data assimilation. Tellus A 56(5):415–428CrossRefGoogle Scholar
  33. Park J (2014) Shrinkage estimator in normal mean vector estimation based on conditional maximum likelihood estimators. Stat Probab Lett 93:1–6CrossRefGoogle Scholar
  34. Ravikumar P, Wainwright MJ, Raskutti G, Yu B (2011) High-dimensional covariance estimation by minimizing L1-penalized log-determinant divergence. Electron J Stat 5:935–980CrossRefGoogle Scholar
  35. Sakov P, Bertino L (2011) Relation between two common localisation methods for the ENKF. Comput Geosci 15(2):225–237CrossRefGoogle Scholar
  36. Song H, Hoteit I, Cornuelle BD, Subramanian AC (2010) An adaptive approach to mitigate background covariance limitations in the ensemble Kalman filter. Mon Weather Rev 138(7):2825–2845CrossRefGoogle Scholar
  37. Sakov P, Oke P (2008) A deterministic formulation of the ensemble Kalman filter: an alternative to ensemble square root filters. Tellus A 60(2)Google Scholar
  38. Whitaker JS, Hamill Thomas M (2002) Ensemble data assimilation without perturbed observations. Mon Weather Rev 16(3):1913–1924CrossRefGoogle Scholar
  39. Zupanski M (2009) Theoretical and practical issues of ensemble data assimilation in weather and climate. In: Park SK, Xu L (eds) Data assimilation for atmospheric, oceanic and hydrologic applications. Springer, Heidelberg, pp 67–84Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.Computational Science Laboratory, Department of Computer ScienceVirginia Polytechnic Institute and State UniversityBlacksburgUSA
  2. 2.Department of Computer ScienceUniversidad del NorteBarranquillaColombia

Personalised recommendations