Efficient parallel implementation of DDDAS inference using an ensemble Kalman filter with shrinkage covariance matrix estimation



This paper develops an efficient and parallel implementation of dynamically data-driven application systems inference using an ensemble Kalman filter based on shrinkage covariance matrix estimation. The proposed implementation works as follows: each model component is surrounded by a local box of radius size r and then, local assimilation steps are carried out in parallel at the different local boxes. Once local analyses are obtained, they are mapped back onto the global domain from which the global analysis state is obtained. Local background error correlations are estimated using the Rao–Blackwell Ledoit and Wolf estimator in order to mitigate the impact of spurious correlations whenever the number of local model components is larger than the ensemble size. The numerical atmospheric general circulation model (SPEEDY) is utilized for the numerical experiments with the T-63 resolution on the BlueRidge cluster at Virginia Tech. The number of processors ranges from 96 to 2048. The proposed implementation outperforms in terms of accuracy the well-known local ensemble transform Kalman filter (LETKF) for all the model variables. The computational time of the proposed implementation is similar to that of the parallel LETKF method (where no covariance estimation is performed) for the largest number of processors.


Parallel EnKF Shrinkage covariance matrix estimation Sampling errors 



This work was supported in part by awards NSF CCF-1218454, AFOSR FA9550-12-1-0293-DEF, and by the Computational Science Laboratory at Virginia Tech.


  1. 1.
    Anderson, J.L.: Localization and sampling error correction in ensemble Kalman filter data assimilation. Mon. Weather Rev. 140(7), 2359–2371 (2012)CrossRefGoogle Scholar
  2. 2.
    Anderson, J.L., Anderson, S.L.: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Weather Rev. 127(12), 2741–2758 (1999)CrossRefGoogle Scholar
  3. 3.
    Anderson, E., Bai, Z., Dongarra, J., Greenbaum, A., McKenney, A., Du Croz, J., Hammerling, S., Demmel, J., Bischof, C., Sorensen, D.: LAPACK: a portable linear algebra library for high-performance computers. In: Proceedings of the 1990 ACM/IEEE Conference on Supercomputing, Supercomputing ’90, pp. 2–11. IEEE Computer Society Press, Los Alamitos (1990)Google Scholar
  4. 4.
    Aved, A., Darema, F., Blasch, E.: Dynamic data driven application systems. (2014)
  5. 5.
    Blackford, L.S., Demmel, J., Dongarra, J., Duff, I., Hammarling, S., Henry, G., Heroux, M., Kaufman, L., Lumsdaine, A., Petitet, A., Pozo, R., Remington, K., Whaley, R.C.: An updated set of basic linear algebra subprograms (BLAS). ACM Trans. Math. Softw. 28, 135–151 (2001)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Blasch, E., Seetharaman, G., Reinhardt, K.: Dynamic data driven applications system concept for information fusion. Proc. Comput. Sci. 18, 1999–2007 (2013). 2013 International Conference on Computational ScienceGoogle Scholar
  7. 7.
    Chen, Y., Wiesel, A., Eldar, Y.C., Hero, A.O.: Shrinkage algorithms for MMSE covariance estimation. IEEE Trans. Signal Process. 58(10), 5016–5029 (2010)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Cheng, H., Jardak, M., Alexe, M., Sandu, A.: A hybrid approach to estimating error covariances in variational data assimilation. Tellus A 62(3), 288–297 (2010)CrossRefGoogle Scholar
  9. 9.
    Cheng, H., Jardak, M., Alexe, M., Sandu, A.: A hybrid approach to estimating error covariances in variational data assimilation. Tellus A 62(3), 288–297 (2010)CrossRefGoogle Scholar
  10. 10.
    Couillet, R., McKay, M.: Large dimensional analysis and optimization of robust shrinkage covariance matrix estimators. J. Multivar. Anal. 131, 99–120 (2014)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Daniels, M.J., Kass, R.E.: Shrinkage estimators for covariance matrices. Biometrics 57(4), 1173–1184 (2001)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Evensen, G.: Data assimilation: the ensemble Kalman filter. Springer, Secaucus (2006)MATHGoogle Scholar
  13. 13.
    Evensen, G.: EnKF—the ensemble Kalman filter. (2015). Accessed 24 Apr 2015
  14. 14.
    Godinez, H.C., Moulton, J.D.: An efficient matrix-free algorithm for the ensemble Kalman filter. Comput. Geosci. 16(3), 565–575 (2012)CrossRefGoogle Scholar
  15. 15.
    Jonathan, P., Fuqing, Z., Weng, Y.: The effects of sampling errors on the EnKF assimilation of inner-core hurricane observations. Mon. Weather Rev. 142(4), 1609–1630 (2014)CrossRefGoogle Scholar
  16. 16.
    Keppenne, C.L.: Data assimilation into a primitive-equation model with a parallel ensemble Kalman filter. Mon. Weather Rev. 128(6), 1971–1981 (2000)CrossRefGoogle Scholar
  17. 17.
    Kucharski, F., Molteni, F., Bracco, A.: Decadal interactions between the western tropical Pacific and the North Atlantic oscillation. Clim. Dynam. 26(1), 79–91 (2006)CrossRefGoogle Scholar
  18. 18.
    Ledoit, O., Wolf, M.: Honey, i shrunk the sample covariance matrix. UPF economics and business working paper (691) (2003)Google Scholar
  19. 19.
    Ledoit, O., Wolf, M.: A well-conditioned estimator for large-dimensional covariance matrices. J. Multivar. Anal. 88(2), 365–411 (2004)MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Lorenc, A.C.: Analysis methods for numerical weather prediction. Q. J. R. Meteorol. Soc. 112(474), 1177–1194 (1986)CrossRefGoogle Scholar
  21. 21.
    Molteni, F.: Atmospheric simulations using a GCM with simplified physical parametrizations. I: model climatology and variability in multi-decadal experiments. Clim. Dynam. 20(2–3), 175–191 (2003)CrossRefGoogle Scholar
  22. 22.
    Nino-Ruiz, E.D., Sandu, A.: An efficient parallel implementation of the ensemble Kalman filter based on shrinkage covariance matrix estimation. In: Proceedings of the 2015 IEEE 22nd International Conference on High Performance Computing Workshops (HiPCW). IEEE Computer Society (2015)Google Scholar
  23. 23.
    Nino-Ruiz, E.D., Sandu, A.: Ensemble Kalman filter implementations based on shrinkage covariance matrix estimation. Ocean Dynam. 65(11), 1423–1439 (2015)CrossRefGoogle Scholar
  24. 24.
    Nino-Ruiz, E.D., Sandu, A., Anderson, J.: An efficient implementation of the ensemble Kalman filter based on an iterative Sherman-Morrison formula. Stat. Comput. 25(3), 561–577 (2015)MathSciNetCrossRefMATHGoogle Scholar
  25. 25.
    Ott, E., Hunt, B.R., Szunyogh, I., Zimin, A.V., Kostelich, Eric J, Corazza, Matteo, Kalnay, Eugenia, Patil, D .J., Yorke, James A: A local ensemble Kalman filter for atmospheric data assimilation. Tellus A 56(5), 415–428 (2004)CrossRefGoogle Scholar
  26. 26.
    Ott, E., Hunt, B., Szunyogh, I., Zimin, A.V., Kostelich, Eic J, Corazza, Matteo, Kalnay, Eugenia, Patil, D .J., Yorke, James A: A local ensemble transform Kalman filter data assimilation system for the NCEP global model. Tellus A 60(1), 113–130 (2008)CrossRefMATHGoogle Scholar
  27. 27.
    Petra, C.G., Zavala, V.M., Nino-Ruiz, E.D., Anitescu, M.: A high-performance computing framework for analyzing the economic impacts of wind correlation. Electr. Power Syst. Res. 141, 372–380 (2016)CrossRefGoogle Scholar
  28. 28.
    Rao, V., Sandu, A.: A posteriori error estimates for DDDAS inference problems. In: Proceedings of the International Conference on Computational Science (ICCS-2014), vol. 29, pp. 1256–1265 (2014)Google Scholar
  29. 29.
    Rao, V., Sandu, A.: A posteriori error estimates for inverse problems. SIAM/ASA J. Uncertain. Quantif. 3(1), 737–761 (2015)MathSciNetCrossRefMATHGoogle Scholar
  30. 30.
    Sakov, P., Bertino, L.: Relation between two common localisation methods for the ENKF. Comput. Geosci. 15(2), 225–237 (2011)CrossRefMATHGoogle Scholar
  31. 31.
    Sandu, A., Constantinescu, E.M., Carmichael, G.R., Chai, T., Daescu, D., Seinfeld, J.H.: Ensemble methods for dynamic data assimilation of chemical observations in atmospheric models. J. Algorithms Comput. Technol. 5(4), 667–692 (2011)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Schäfer, J., Strimmer, K., et al.: A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Stat. Appl. Genet. Mol. Biol. 4(1), 32 (2005)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Xiaohui, C., Wang, Z.J., McKeown, M.J.: Shrinkage-to-tapering estimation of large covariance matrices. IEEE Trans. Signal Process. 60(11), 5640–5656 (2012)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Zupanski, M.: Theoretical and practical issues of ensemble data assimilation in weather and climate. In: Park, S.K., Xu, L. (eds.) Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications, pp. 67–84. Springer, Berlin, Heidelberg (2009)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversidad del NorteBarranquillaColombia
  2. 2.Computational Science Laboratory, Department of Computer ScienceVirginia TechBlacksburgUSA

Personalised recommendations