Abstract
In this paper, we propose an efficient and practical implementation of the ensemble Kalman filter via shrinkage covariance matrix estimation. Our filter implementation combines information brought by an ensemble of model realizations, and that based on our prior knowledge about the dynamical system of interest. We perform the combination of both sources of information via optimal shrinkage factors. The method exploits the rank-deficiency of ensemble covariance matrices to provide an efficient and practical implementation of the analysis step in EnKF based formulations. Localization and inflation aspects are discussed, as well. Experimental tests are performed to assess the accuracy of our proposed filter implementation by employing an Advection Diffusion Model and an Atmospheric General Circulation Model. The experimental results reveal that the use of our proposed filter implementation can mitigate the impact of sampling noise, and even more, it can avoid the impact of spurious correlations during assimilation steps.
Article PDF
Similar content being viewed by others
References
Anderson, J.L.: An Ensemble Adjustment Kalman Filter for Data Assimilation. Monthly Weather Review. https://doi.org/10.1175/1520-0493(2001)129<2884:AEAKFF>2.0.CO;2 (2001)
Anderson, J.L., Anderson, S.L.: A Monte Carlo Implementation of the Nonlinear Filtering Problem to Produce Ensemble Assimilations and Forecasts. Monthly Weather Review. https://doi.org/10.1175/1520-0493(1999)127<2741:AMCIOT>2.0.CO;2 (1999)
Asch, M., Bocquet, M., Nodet, M.: Data assimilation. Society for industrial and applied mathematics, Philadelphia. https://doi.org/10.1137/1.9781611974546 (2016)
Barbu, A.L., Segers, A.J., Schaap, M., Heemink, A.W., Builtjes, P.J.H.: A multi-component data assimilation experiment directed to sulphur dioxide and sulphate over Europe. Atmos. Environ. 43(9), 1622–1631 (2009). https://doi.org/10.1016/j.atmosenv.2008.12.005
Bracco, A., Kucharski, F., Kallummal, R., Molteni, F.: Internal variability, external forcing and climate trends in multi-decadal AGCM ensembles. Clim. Dynam. 23(6), 659–678 (2004)
Chen, Y., Wiesel, A., Eldar, Y.C., Hero, A.O.: Shrinkage algorithms for MMSE covariance estimation. IEEE Trans. Signal Process. 58(10), 5016–5029 (2010). https://doi.org/10.1109/TSP.2010.2053029
Chen, Y.H., Prinn, R.G.: Estimation of atmospheric methane emissions between 1996 and 2001 using a three-dimensional global chemical transport model. J. Geophys. Res. Atmosph. 111(10), 1–25 (2006). https://doi.org/10.1029/2005JD006058
Couillet, R., McKay, M.: Large dimensional analysis and optimization of robust shrinkage covariance matrix estimators. J. Multivar. Anal. 131, 99–120 (2014)
Evensen, G.: The Ensemble Kalman Filter: Theoretical formulation and practical implementation. Ocean Dyn. 53(4), 343–367 (2003). https://doi.org/10.1007/s10236-003-0036-9
Fu, G., Prata, F., Xiang lin, H., Heemink, A., Segers, A., Lu, S.: Data assimilation for volcanic ash plumes using a satellite observational operator: A case study on the 2010 Eyjafjallajȯkull volcanic eruption. Atmos. Chem. Phys. 17(2), 1187–1205 (2017). https://doi.org/10.5194/acp-17-1187-2017
Gaspari, G., Cohn, S.E.: Construction of correlation functions in two and three dimensions. Q. J. Roy. Meteorol. Soc. 125(554), 723–757 (1999). https://doi.org/10.1002/qj.49712555417
Greybush, S.J., Kalnay, E., Miyoshi, T., Ide, K., Hunt, B.R.: Balance and Ensemble Kalman Filter Localization Techniques. Mon. Weather. Rev. 139(2), 511–522 (2011). https://doi.org/10.1175/2010MWR3328.1
Hamill, T.M., Whitaker, J.S., Snyder, C.: Distance-Dependent Filtering of Background Error Covariance Estimates in an Ensemble Kalman Filter. Monthly Weather Review. https://doi.org/10.1175/1520-0493(2001)129<2776:DDFOBE>2.0.CO;2 (2001)
Houtekamer, P., Mitchell, H.: A Sequential Ensemble Kalman Filter for Atmospheric Data Assimilation. Amer. Meteorol. Soc. 129(ii), 123–137. https://doi.org/10.1175/1520-0493(2001)129<0123:ASEKFF>2.0.CO;2 (2001)
Houtekamer, P.L., Zhang, F.: Review of the Ensemble Kalman Filter for Atmospheric Data Assimilation. monthly weather review. https://doi.org/10.1175/MWR-D-15-0440.1 (2016)
Hunt, B.R., Kostelich, E.J., Szunyogh, I.: Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter. Physica D: Nonlinear Phenom. 230(1-2), 112–126 (2007). https://doi.org/10.1016/j.physd.2006.11.008
Jin, J., Lin, H.X., Heemink, A., Segers, A.: Spatially varying parameter estimation for dust emissions using reduced-tangent-linearization 4DVar. Atmos. Environ. 187, 358–373 (2018). https://doi.org/10.1016/j.atmosenv.2018.05.060
Kalnay, E., Li, H., Miyoshi, T., Yang, S.C., Ballabrera-poy, J.: 4dvar or ensemble kalman filter? Tellus A: Dyn. Meteorol. Oceanogr. 59(5), 758–773 (2007)
Kucharski, F., Molteni, F., Bracco, A.: Decadal interactions between the western tropical pacific and the north atlantic oscillation. Clim. Dyn. 26(1), 79–91 (2006)
Ledoit, O., Wolf, M.: A well-conditioned estimator for large-dimensional covariance matrices. J. Multivar. Anal. 88(2), 365–411 (2004)
Ledoit, O., Wolf, M.: A well-conditioned estimator for large-dimensional covariance matrices. J. Multivar. Anal. 88(2), 365–411. https://doi.org/10.1016/S0047-259X(03)00096-4 (2004)
Ledoit, O., Wolf, M., et al.: Optimal estimation of a large-dimensional covariance matrix under stein’s loss. Bernoulli 24(4B), 3791–3832 (2018)
Lopez-Restrepo, S., Yarce, A., Pinel, N., Quintero, O., Segers, A., Heemink, A.: Forecasting PM10 and PM2.5 in the Aburrá Valley (Medellín, Colombia) via EnKF based Data Assimilation. Atmospheric Environment. Accepted (2020)
Lu, S., Lin, H.X., Heemink, A.W., Fu, G., Segers, A.J.: estimation of volcanic ash emissions using Trajectory-Based 4D-Var data assimilation. Mon. Weather. Rev. 144(2), 575–589 (2016). https://doi.org/10.1175/MWR-D-15-0194.1
Miyoshi, T.: The gaussian approach to adaptive covariance inflation and its implementation with the local ensemble transform kalman filter. Mon. Weather. Rev. 139(5), 1519–1535 (2011)
Molteni, F.: Atmospheric simulations using a gcm with simplified physical parametrizations. i: Model climatology and variability in multi-decadal experiments. Climate Dynam. 20(2-3), 175–191 (2003)
Nan, T., Wu, J.: Groundwater parameter estimation using the ensemble Kalman filter with localization. Hydrogeol. J. 19(3), 547–561 (2011). https://doi.org/10.1007/s10040-010-0679-9
Nino-Ruiz, E.D., Guzman, L., Jabba, D.: An ensemble kalman filter implementation based on the ledoit and wolf covariance matrix estimator. J. Comput. Appl. Math. 384, 113163. https://doi.org/10.1016/j.cam.2020.113163(2021)
Nino-Ruiz, E.D., Guzman-Reyes, L.G., Beltran-Arrieta, R.: An adjoint-free four-dimensional variational data assimilation method via a modified cholesky decomposition and an iterative woodbury matrix formula. Nonlinear Dyn. 99(3), 2441–2457 (2020)
Nino-Ruiz, E.D., Sandu, A.: Ensemble kalman filter implementations based on shrinkage covariance matrix estimation. Ocean Dyn. 65(11), 1423–1439 (2015)
Nino-Ruiz, E.D., Sandu, A.: Efficient parallel implementation of dddas inference using an ensemble kalman filter with shrinkage covariance matrix estimation. Clust. Comput., 1–11 (2017)
Ott, E., Hunt, B.R., Szunyogh, I., Zimin, A.V., Kostelich, E., Corazza, M., Kalnay, E., Patil, D., Yorke, J.A.: A local ensemble Kalman filter for atmospheric data assimilation. Tellus 56, 415–428 (2004)
Petrie, R.E.: Localization in the Ensemble Kalman Filter. Master, University of Reading (2008)
Quintero, M.O.L., Amicarelli, A.A., Scaglia, G., di Sciascio, F.: Control based on numerical methods and recursive bayesian estimation in a continuous alcoholic fermentation process. BioResources 4(4), 1372–1395 (2009)
Quintero, M.O.L., Scaglia, G., Di Sciascio, F., Mut, V.: Numerical Methods Based Strategy and Particle Filter State Estimation for Bio Process Control. In: 2008 IEEE International Conference on Industrial Technology, pp. 1–6. IEEE (2008)
Richardson, P.L., Mooney, K.: The mediterranean outflow—a simple advection-diffusion model. J. Phys. Oceanogr. 5(3), 476–482. https://doi.org/10.1175/1520-0485(1975)005<0476:TMOSAD>2.0.CO;2 (1975)
Sakov, P., Bertino, L.: Relation between two common localisation methods for the enKF. Comput. Geosci. 15(2), 225–237 (2011). https://doi.org/10.1007/s10596-010-9202-6
Sakov, P., Evensen, G., Bertino, L.: Asynchronous data assimilation with the enKF. Tellus, Ser. A: Dyn. Meteorol. Oceanogr. 62(1), 24–29 (2010). https://doi.org/10.1111/j.1600-0870.2009.00417.x
Stoica, P., Li, J., Zhu, X., Guerci, J.R.: On using a priori knowledge in space-time adaptive processing. IEEE Trans. Signal Process. 56(6), 2598–2602 (2008). https://doi.org/10.1109/TSP.2007.914347
Timmermans, R., Segers, A., Curier, L., Abida, R., Attiė, J.L., El Amraoui, L., Eskes, H., De Haan, J., Kujanpȧȧ, J., Lahoz, W., Oude Nijhuis, A., Quesada-Ruiz, S., Ricaud, P., Veefkind, P., Schaap, M.: Impact of synthetic space-borne NO2 observations from the Sentinel-4 and Sentinel-5P missions on tropospheric NO2 analyses. Atmos. Chem. Phys. 19(19), 12811–12833 (2019). https://doi.org/10.5194/acp-19-12811-2019
Tirabassi, T.: Analytical air pollution advection and diffusion models. Water Air Soil Pollut. 47 (1), 19–24 (1989). https://doi.org/10.1007/BF00468993
Touloumis, A.: Nonparametric stein-type shrinkage covariance matrix estimators in high-dimensional settings. Comput. Stat. Data Anal. 83, 251–261 (2015)
Wang, X., Barker, D.M., Snyder, C., Hamill, T.M.: A hybrid etkf–3dvar data assimilation scheme for the wrf model. part i: Observing system simulation experiment. Mon. Weather. Rev. 136(12), 5116–5131 (2008)
Wang, X., Snyder, C., Hamill, T.M.: On the theoretical equivalence of differently proposed ensemble–3dvar hybrid analysis schemes. Mon. Weather. Rev. 135(1), 222–227 (2007)
Zhu, X., Li, J., Stoica, P.: Knowledge-Aided Space-Time adaptive processing. IEEE Trans. Aerosp. Electron. Syst. 47(2), 1325–1336 (2011)
Zhu, Y., Toth, Z., Wobus, R., Richardson, D., Mylne, K.: The Economic Value of Ensemble-Based Weather Forecasts. Bullet. Amer. Meteorol. Soci. 83(1), 73–84. https://doi.org/10.1175/1520-0477(2002)083<0073:TEVOEB>2.3.CO;2 (2002)
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Lopez-Restrepo, S., Nino-Ruiz, E.D., Guzman-Reyes, L.G. et al. An efficient ensemble Kalman Filter implementation via shrinkage covariance matrix estimation: exploiting prior knowledge. Comput Geosci 25, 985–1003 (2021). https://doi.org/10.1007/s10596-021-10035-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10596-021-10035-4
Keywords
- Data assimilation
- Air quality
- Chemical transport model
- Ensemble Kalman Filter
- Background error covariance matrix