Abstract
Ensemble Kalman filters are based on a Gaussian assumption, which can limit their performance in some non-Gaussian settings. This paper reviews two nonlinear, non-Gaussian extensions of the Ensemble Kalman Filter: Gaussian anamorphosis (GA) methods and two-step updates, of which the rank histogram filter (RHF) is a prototypical example. GA-EnKF methods apply univariate transforms to the state and observation variables to make their distribution more Gaussian before applying an EnKF. The two-step methods use a scalar Bayesian update for the first step, followed by linear regression for the second step. The connection of the two-step framework to the full Bayesian problem is made, which opens the door to more advanced two-step methods in the full Bayesian setting. A new method for the first part of the two-step framework is proposed, with a similar form to the RHF but a different motivation, called the ‘improved RHF’ (iRHF). A suite of experiments with the Lorenz-‘96 model demonstrate situations where the GA-EnKF methods are similar to EnKF, and where they outperform EnKF. The experiments also strongly support the accuracy of the RHF and iRHF filters for nonlinear and non-Gaussian observations; these methods uniformly beat the EnKF and GA-EnKF methods in the experiments reported here. The new iRHF method is only more accurate than RHF at small ensemble sizes in the experiments reported here.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
PART II: DATA ASSIMILATION, chap. 2. Integrated Forecasting System Documentation. ECMWF. https://www.ecmwf.int/node/19306 (2019)
Aanonsen, S.I., Nævdal, G., Oliver, D.S., Reynolds, A.C., Vallè, s, B., et al.: The ensemble Kalman filter in reservoir engineering–a review. Spe J. 14(03), 393–412 (2009)
Ades, M., Van Leeuwen, P.J.: An exploration of the equivalent weights particle filter. Q. J. Roy. Meteor. Soc. 139(672), 820–840 (2013)
Ades, M., Van Leeuwen, P.J.: The equivalent-weights particle filter in a high-dimensional system. Q. J. Roy. Meteor. Soc. 141(687), 484–503 (2015)
Albers, D.J., Blancquart, P.A., Levine, M.E., Seylabi, E.E., Stuart, A.: Ensemble Kalman methods with constraints. Inverse Problems 35(9), 095007 (2019)
Amezcua, J., Van Leeuwen, P.J.: Gaussian anamorphosis in the analysis step of the enKF: a joint state-variable/observation approach. Tellus A 66(1), 23493 (2014)
Anderson, J.L.: An ensemble adjustment Kalman filter for data assimilation. Mon. Weather Rev. 129(12), 2884–2903 (2001)
Anderson, J.L.: A local least squares framework for ensemble filtering. Mon. Weather Rev. 131 (4), 634–642 (2003)
Anderson, J.L.: A non-Gaussian ensemble filter update for data assimilation. Mon. Weather Rev. 138(11), 4186–4198 (2010)
Anderson, J.L.: A nonlinear rank regression method for ensemble Kalman filter data assimilation. Mon. Weather Rev. 147(8), 2847–2860 (2019)
Anderson, J.L.: A marginal adjustment rank histogram filter for non-Gaussian ensemble data assimilation. Mon. Weather Rev. 148(8), 3361–3378 (2020)
Anderson, J.L., Anderson, S.L.: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Weather Rev. 127(12), 2741–2758 (1999)
Attia, A., Moosavi, A., Sandu, A.: Cluster sampling filters for non-gaussian data assimilation. Atmosphere 9(6), 213 (2018)
Bannister, R.: A review of operational methods of variational and ensemble-variational data assimilation. Q. J. Roy. Meteor. Soc. 143(703), 607–633 (2017)
Beal, D., Brasseur, P., Brankart, J.M., Ourmieres, Y., Verron, J.: Characterization of mixing errors in a coupled physical biogeochemical model of the North atlantic: Implications for nonlinear estimation using Gaussian anamorphosis. Ocean Sci. 6, 247–262 (2010)
Bengtsson, T., Bickel, P., Li, B.: Curse-of-dimensionality revisited: Collapse of the particle filter in very large scale systems, Collections, vol. Volume 2, pp. 316–334. Institute of Mathematical Statistics, Beachwood, Ohio USA. https://doi.org/10.1214/193940307000000518 (2008)
Bengtsson, T., Snyder, C., Nychka, D.: Toward a nonlinear ensemble filter for high-dimensional systems. J. Geophys. Res.-Atmos 108(D24) (2003)
Bertino, L., Evensen, G., Wackernagel, H.: Sequential data assimilation techniques in oceanography. Int. Stat. Rev. 71(2), 223–241 (2003)
Bishop, C.H.: The GIGG-enKF: ensemble Kalman filtering for highly skewed non-negative uncertainty distributions. Q. J. Roy. Meteor. Soc. 142(696), 1395–1412 (2016)
Bishop, C.H., Etherton, B.J., Majumdar, S.J.: Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Mon. Weather Rev. 129(3), 420–436 (2001)
Bishop, C.H., Whitaker, J.S., Lei, L.: Gain form of the ensemble transform Kalman filter and its relevance to satellite data assimilation with model space ensemble covariance localization. Mon. Weather Rev. 145(11), 4575–4592 (2017)
Bocquet, M., Pires, C.A., Wu, L.: Beyond Gaussian statistical modeling in geophysical data assimilation. Mon. Weather Rev. 138(8), 2997–3023 (2010)
Brankart, J.M., Testut, C.E., Béal, D., Doron, M., Fontana, C., Meinvielle, M., Brasseur, P., Verron, J.: Towards an improved description of ocean uncertainties: Effect of local anamorphic transformations on spatial correlations. Ocean Sci. 8(2), 121 (2012)
Brusdal, K., Brankart, J.M., Halberstadt, G., Evensen, G., Brasseur, P., van Leeuwen, P.J., Dombrowsky, E., Verron, J.: A demonstration of ensemble-based assimilation methods with a layered OGCM from the perspective of operational ocean forecasting systems. J.Marine Sys. 40, 253–289 (2003)
Burgers, G., van Leeuwen, P., Evensen, G.: Analysis scheme in the ensemble Kalman filter. Mon. Weather Rev. 126(6), 1719–1724 (1998)
Chorin, A., Morzfeld, M., Tu, X.: Implicit particle filters for data assimilation. Comm App Math Com Sc 5(2), 221–240 (2010)
Chorin, A.J., Tu, X.: Implicit sampling for particle filters. Proc. Natl. Acad. Sci. USA 106 (41), 17249–17254 (2009)
Chorin, A.J., Tu, X.: An iterative implementation of the implicit nonlinear filter. ESAIM-Math Model Num. 46(3), 535–543 (2012)
Chustagulprom, N., Reich, S., Reinhardt, M.: A hybrid ensemble transform particle filter for nonlinear and spatially extended dynamical systems. SIAM-ASA. J. Uncertainty Quant. 4(1), 592–608 (2016)
Crisan, D., Doucet, A.: A survey of convergence results on particle filtering methods for practitioners. IEEE T signal proces 50(3), 736–746 (2002)
Doron, M., Brasseur, P., Brankart, J.M., Losa, S.N., Melet, A.: Stochastic estimation of biogeochemical parameters from Globcolour ocean colour satellite data in a North Atlantic 3D ocean coupled physical–biogeochemical model. J.Marine Systems 117, 81–95 (2013)
Doucet, A., De Freitas, N., Gordon, N.: An Introduction to Sequential Monte Carlo Methods. In: Sequential Monte Carlo Methods in Practice, pp 3–14. Springer (2001)
Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res.-Oceans 99(C5), 10143–10162 (1994)
Evensen, G.: Data assimilation: the ensemble kalman filter springer (2009)
Fischer, C., Montmerle, T., Berre, L., Auger, L., ŞtefĂnescu, S.E.: An overview of the variational assimilation in the ALADIN/france numerical weather-prediction system. Q. J. Roy. Meteor. Soc. 131 (613), 3477–3492 (2005)
Fossum, K., Mannseth, T., Stordal, A.S.: Assessment of multilevel ensemble-based data assimilation for reservoir history matching. Computat. Geosci. 24(1), 217–239 (2020)
Frei, M., Künsch, H.R.: Bridging the ensemble Kalman and particle filters. Biometrika 100 (4), 781–800 (2013)
Frei, M., Künsch, H.R.: Mixture ensemble Kalman filters. Comput.Stat.Data An. 58, 127–138 (2013)
Gharamti, M., Samuelsen, A., Bertino, L., Simon, E., Korosov, A., Daewel, U.: Online tuning of ocean biogeochemical model parameters using ensemble estimation techniques: Application to a one-dimensional model in the North Atlantic. J.Marine Systems 168, 1–16 (2017)
Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. JAmStatAssoc 102, 359–378 (2007)
Gopalakrishnan, G., Cornuelle, B.D., Hoteit, I., Rudnick, D.L., Owens, W.B.: State estimates and forecasts of the loop current in the Gulf of Mexico using the MITgcm and its adjoint. J. Geophys. Res.-Oceans 118(7), 3292–3314 (2013)
Gordon, N.J., Salmond, D.J., Smith, A.F.: Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In: IEE Proceedings F (Radar and Signal Processing), vol. 140, pp 107–113. IET (1993)
Grooms, I.: A note on the formulation of the Ensemble Adjustment Kalman Filter. arXiv:2006.02941 (2020)
Grooms, I.: iangrooms/Nonlinear-EnKF: Nonlinear EnKF Methods in Lorenz-96. https://doi.org/10.5281/zenodo.5889403. https://github.com/iangrooms/Nonlinear-EnKF (2022)
Grooms, I., Robinson, G.: A hybrid particle-ensemble Kalman filter for problems with medium nonlinearity. PLoS ONE 16, e0248266 (2021)
Hersbach, H.: Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast 15, 559–570 (2000)
Hodyss, D.: Accounting for skewness in ensemble data assimilation. Mon. Weather Rev. 140 (7), 2346–2358 (2012)
Hodyss, D., Anderson, J.L., Collins, N., Campbell, W.F., Reinecke, P.A.: Quadratic polynomial regression using serial observation processing: Implementation within DART. Mon. Weather Rev. 145 (11), 4467–4479 (2017)
Hoteit, I., Pham, D.T., Triantafyllou, G., Korres, G.: A new approximate solution of the optimal nonlinear filter for data assimilation in meteorology and oceanography. Mon. Weather Rev. 136(1), 317–334 (2008)
Houtekamer, P., Mitchell, H.: Data assimilation using an ensemble Kalman filter technique. Mon. Weather Rev. 126(3), 796–811 (1998)
Houtekamer, P.L., Zhang, F.: Review of the ensemble Kalman filter for atmospheric data assimilation. Mon. Weather Rev. 144(12), 4489–4532 (2016)
Hu, C.C., van Leeuwen, P.J.: A particle flow filter for high-dimensional system applications. Q. J. Roy. Meteor. Soc. 147, 2352–2374 (2021)
Huang, X.Y., Xiao, Q., Barker, D.M., Zhang, X., Michalakes, J., Huang, W., Henderson, T., Bray, J., Chen, Y., Ma, Z., et al.: Four-dimensional variational data assimilation for WRF: formulation and preliminary results. Mon. Weather Rev. 137(1), 299–314 (2009)
Hunt, B.R., Kostelich, E.J., Szunyogh, I.: Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter. Physica D 230(1-2), 112–126 (2007)
Iglesias, M.A., Law, K.J., Stuart, A.M.: Evaluation of gaussian approximations for data assimilation in reservoir models. Computat. Geosci. 17(5), 851–885 (2013)
Janjić, T., McLaughlin, D., Cohn, S.E., Verlaan, M.: Conservation of mass and preservation of positivity with ensemble-type Kalman filter algorithms. Mon. Weather Rev. 142(2), 755–773 (2014)
Jazwinski, A.H.: Stochastic processes and filtering theory academic press (1970)
Kalnay, E.: Atmospheric modeling, data assimilation, and predictability. Cambridge University Press (2002)
Kotsuki, S., Miyoshi, T., Terasaki, K., Lien, G.Y., Kalnay, E.: Assimilating the global satellite mapping of precipitation data with the Nonhydrostatic Icosahedral Atmospheric Model (NICAM). J. Geophys. Res. Atmos. 122(2), 631–650 (2017)
Kwiatkowski, E., Mandel, J.: Convergence of the square root ensemble Kalman filter in the large ensemble limit. SIAM-ASA. J. Uncertainty Quant. 3(1), 1–17 (2015)
Lauvernet, C., Brankart, J.M., Castruccio, F., Broquet, G., Brasseur, P., Verron, J.: A truncated Gaussian filter for data assimilation with inequality constraints: Application to the hydrostatic stability condition in ocean models. Ocean Model. 27(1-2), 1–17 (2009)
Law, K., Stuart, A., Zygalakis, K.: Data assimilation a mathematical introduction springer heidelberg (2015)
Law, K.J., Stuart, A.M.: Evaluating data assimilation algorithms. Mon. Weather Rev. 140 (11), 3757–3782 (2012)
Lei, J., Bickel, P., Snyder, C.: Comparison of ensemble Kalman filters under non-Gaussianity. Mon. Weather Rev. 138(4), 1293–1306 (2010)
Li, L., Zhou, H., Hendricks Franssen, H.J., Gómez-Hernández, J.J.: Groundwater flow inverse modeling in non-MultiGaussian media: performance assessment of the normal-score Ensemble Kalman Filter. Hydrol. Earth Syst. Sci. 16(2), 573–590 (2012)
Li, R., Jan, N.M., Huang, B., Prasad, V.: Constrained ensemble Kalman filter based on Kullback–Leibler divergence. J.Process Control 81, 150–161 (2019)
Lien, G.Y., Kalnay, E., Miyoshi, T.: Effective assimilation of global precipitation: Simulation experiments. Tellus A 65(1), 19915 (2013)
Lien, G.Y., Kalnay, E., Miyoshi, T., Huffman, G.J.: Statistical properties of global precipitation in the NCEP GFS model and TMPA observations for data assimilation. Mon. Weather Rev. 144(2), 663–679 (2016)
Lien, G.Y., Miyoshi, T., Kalnay, E.: Assimilation of TRMM multisatellite precipitation analysis with a low-resolution NCEP global forecast system. Mon. Weather Rev. 144(2), 643–661 (2016)
Liu, B., Ait-El-Fquih, B., Hoteit, I.: Efficient kernel-based ensemble Gaussian mixture filtering. Mon. Weather Rev. 144(2), 781–800 (2016)
Lorenz, E.: Predictability: a problem partly solved. In: Proceedings of Seminar on Predicability, vol. 1, pp 1–18. ECMWF, Reading, UK (1996)
Lorenz, E.N., Emanuel, K.A.: Optimal sites for supplementary weather observations: Simulation with a small model. J. Atmos. Sci. 55(3), 399–414 (1998)
Mandel, J., Cobb, L., Beezley, J.D.: On the convergence of the ensemble Kalman filter. Appl. Math. 56(6), 533–541 (2011)
Marzouk, Y., Moselhy, T., Parno, M., Spantini, A.: Sampling via Measure Transport: An Introduction, pp 1–41. Springer International Publishing, Cham (2016). https://doi.org/10.1007/978-3-319-11259-6_23-1
Metref, S., Cosme, E., Snyder, C., Brasseur, P.: A non-Gaussian analysis scheme using rank histograms for ensemble data assimilation. Nonlinear Proc. Geoph. 21(4), 869–885 (2014)
Moore, A.M., Arango, H.G., Broquet, G., Powell, B.S., Weaver, A.T., Zavala-Garay, J.: The Regional Ocean Modeling System (ROMS) 4-dimensional variational data assimilation systems: Part I–system overview and formulation. Prog. Oceanogr. 91(1), 34–49 (2011)
Morzfeld, M., Hodyss, D., Poterjoy, J.: Variational particle smoothers and their localization. Q. J. Roy. Meteor. Soc. 144(712), 806–825 (2018)
Nelson, R.B.: An introduction to copulas, second edn. Springer (2006)
Nino-Ruiz, E.D., Cheng, H., Beltran, R.: A robust non-Gaussian data assimilation method for highly non-linear models. Atmosphere 9(4), 126 (2018)
Penny, S.G., Miyoshi, T.: A local particle filter for high-dimensional geophysical systems. Nonlinear Proc. Geoph. 23(6), 391–405 (2016)
Pham, D.T., Verron, J., Gourdeau, L.: Singular evolutive Kalman filters for data assimilation in oceanography. Comptes Rendus de l’Academie des Sciences Series IIA Earth and Planetary Science 4 (326), 255–260 (1998)
Pinheiro, F.R., van Leeuwen, P.J., Geppert, G.: Efficient nonlinear data assimilation using synchronization in a particle filter. Q. J. Roy. Meteor. Soc. 145(723), 2510–2523 (2019)
Posselt, D.J., Bishop, C.H.: Nonlinear data assimilation for clouds and precipitation using a gamma inverse-gamma ensemble filter. Q. J. Roy. Meteor. Soc. 144(716), 2331–2349 (2018)
Poterjoy, J.: A localized particle filter for high-dimensional nonlinear systems. Mon. Weather Rev. 144(1), 59–76 (2016)
Poterjoy, J., Wicker, L., Buehner, M.: Progress toward the application of a localized particle filter for numerical weather prediction. Mon. Weather Rev. 147(4), 1107–1126 (2019)
Pourahmadi, M.: Covariance estimation: the GLM and regularization perspectives. Stat. Sci. pp. 369–387 (2011)
Pulido, M., van Leeuwen, P.J.: Sequential Monte Carlo with kernel embedded mappings: The mapping particle filter. J. Comput. Phys. (2019)
Rawlins, F., Ballard, S., Bovis, K., Clayton, A., Li, D., Inverarity, G., Lorenc, A., Payne, T.: The Met Office global four-dimensional variational data assimilation scheme. Q. J. Roy. Meteor. Soc. 133(623), 347–362 (2007)
Rebeschini, P., Van Handel, R.: Can local particle filters beat the curse of dimensionality? Ann Appl Probab 25(5), 2809–2866 (2015)
Robert, S., Künsch, H.R.: Localizing the ensemble Kalman particle filter. Tellus A 69(1), 1282016 (2017)
Robinson, G., Grooms, I., Kleiber, W.: Improving particle filter performance by smoothing observations. Mon. Weather Rev. 146(8), 2433–2446 (2018)
Schölzel, C., Friederichs, P.: Multivariate non-normally distributed random variables in climate research – introduction to the copula approach. Nonlinear Proc. Geoph. 15(5), 761–772 (2008)
Sesterhenn, J.L.: Adjoint-based Data Assimilation of an Epidemiology Model for the Covid-19 Pandemic in 2020. arXiv e-prints (2020)
Silverman, B.: Density estimation for statistics and data analysis CRC Press (1998)
Simon, E., Bertino, L.: Gaussian anamorphosis extension of the DEnKF for combined state parameter estimation: Application to a 1D ocean ecosystem model. J.Marine Systems 89(1), 1–18 (2012)
Sklar, A.: Fonctions de répartition à n dimensions et leurs marges. Publ. Inst. Statist. Univ. Paris 8, 229–231 (1959)
Slivinski, L.C., Compo, G.P., Whitaker, J.S., Sardeshmukh, P.D., Giese, B.S., McColl, C., Allan, R., Yin, X., Vose, R., Titchner, H., et al.: Towards a more reliable historical reanalysis: Improvements for version 3 of the Twentieth Century Reanalysis system. Q. J. Roy. Meteor. Soc. 145(724), 2876–2908 (2019)
Snyder, C., Bengtsson, T., Bickel, P., Anderson, J.: Obstacles to high-dimensional particle filtering. Mon. Weather Rev. 136(12), 4629–4640 (2008)
Snyder, C., Bengtsson, T., Morzfeld, M.: Performance bounds for particle filters using the optimal proposal. Mon. Weather Rev. 143(11), 4750–4761 (2015)
Spantini, A., Baptista, R., Marzouk Y.: Coupling techniques for nonlinear ensemble filtering (2019)
Stordal, A.S., Karlsen, H.A., Nævdal, G., Skaug, H.J., Vallè, s, B.: Bridging the ensemble Kalman filter and particle filters: the adaptive Gaussian mixture filter. Computat. Geosci. 15(2), 293–305 (2011)
Talagrand, O.: Variational Assimilation. In: Lahoz, W. , Khattatov, B. , Menard, R. (eds.) Data Assimilation Making Sense of Observations, pp 41–67. Springer (2010)
Van Leeuwen, P.J.: Particle filtering in geophysical systems. Mon. Weather Rev. 137(12), 4089–4114 (2009)
Wang, X., Bishop, C.H., Julier, S.J.: Which is better, an ensemble of positive–negative pairs or a centered spherical simplex ensemble? Mon. Weather Rev. 132(7), 1590–1605 (2004)
Wang, X., Lei, T.: GSI-Based four-dimensional ensemble–variational (4DEnsvar) data assimilation: Formulation and single-resolution experiments with real data for NCEP Global Forecast System. Mon. Weather Rev. 142(9), 3303–3325 (2014)
Whitaker, J.S., Hamill, T.M.: Ensemble data assimilation without perturbed observations. Mon. Weather Rev. 130(7), 1913–1924 (2002)
Wikle, C.K., Berliner, L.M.: A Bayesian tutorial for data assimilation. Physica D 230(1), 1–16 (2007)
Zhou, H., Gomez-Hernandez, J.J., Franssen, H.J.H., Li, L.: An approach to handling non-Gaussianity of parameters and state variables in ensemble Kalman filtering. Adv. Water Resour. 34(7), 844–864 (2011)
Acknowledgements
The author is grateful to J. L. Anderson for discussions on the two-step framework and the RHF, and to M. El Gharamti for discussions of Gaussian anamorphosis methods. Two anonymous reviewers offered suggestions for clarification and improvement of the presentation. This work utilized resources from the University of Colorado Boulder Research Computing Group, which is supported by the National Science Foundation (awards ACI-1532235 and ACI-1532236), the University of Colorado Boulder, and Colorado State University.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
The author declares that he has no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix: : Copulas
Appendix: : Copulas
This appendix provides a brief, incomplete background on copulas. For further details, see [78, 92]. Sklar’s theorem [96] relates the joint cdf H(x1,…,xd) of a vector of random variables to the marginal cdfs Fk(xk) and a ‘copula’ C:
The copula C is thus a function from the d-dimensional hypercube to [0, 1]; it is uniquely defined when the marginal cdfs are continuous. The copula encodes the dependence relationships between the variables, in the sense that strictly-increasing transformations of the random variables Xi do not change the copula [78, Theorem 2.3.4]. To give an example relevant to the discussion of GA methods, let X1,…,Xd be jointly Gaussian random variables with zero mean, unit variance, and covariance matrix C. Let G1,…,Gd be arbitrary invertible cdfs, and suppose that we apply a probability integral transform to each Xk, resulting in the new set of random variables \(\hat {X}_{k} = G_{k}^{-1}({\varPhi }(X_{k}))\), where Φ is the standard normal cdf. This new set of random variables \(\hat {X}_{1},\ldots ,\hat {X}_{d}\) is clearly not jointly normal (the marginal pdf of \(\hat {X}_{k}\) is \(G_{k}^{\prime }(x)\), which is not Gaussian), but the dependence structure between the variables is completely described by a Gaussian copula. There is a huge menagerie of families of copulas describing different kinds of dependence structure between variables. By transforming all the marginal distributions to Gaussian distributions and then applying the EnKF, which assumes that the joint distribution is Gaussian, GA methods are evidently assuming that the distribution of the original variables has a Gaussian copula.
Rights and permissions
About this article
Cite this article
Grooms, I. A comparison of nonlinear extensions to the ensemble Kalman filter. Comput Geosci 26, 633–650 (2022). https://doi.org/10.1007/s10596-022-10141-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10596-022-10141-x